Skip to content

security-policy.md

Latest commit

 

History

History
410 lines (359 loc) · 12.5 KB

security-policy.md

File metadata and controls

410 lines (359 loc) · 12.5 KB

NSX Operator Security Policy CRD

Summary

Standard K8s NetworkPolicies secures traffic between Pods, a cluster administrator may want to have the ability to configure consistent network policy through a common API for Namespaces which apply to both Pods and VMs in K8s cluster. This document describes SecurityPolicy CRD to support the security control of workloads within the cluster. SecurityPolicy CRD is used to apply NSX-T based security policy to VMs and Pods in K8s cluster. VMs should be defined as workload resources in K8s cluster, both VMs and Pods are attached to NSX-T segments, each workload has segment port on NSX-T. Labels on VMs and Pods are tagged on their segment ports. In the first version, nsx-operator leverages Tags added by NSX Container Plugin(NCP) on NSX-T segment ports, nsx-operator needs to work together with NCP on Supervisor cluster of vSphere with Kubernetes. nsx-operator will reconcile SecurityPolicy CRs, call NSX-T API to create NSX-T Distributed Firewall rules, then update CR status with realized state.

CRD Design

SecurityPolicy is Namespaced scope CRD, admins could apply CR within a namespace to configure security of workloads which are attached to NSX-T networking. Compared with standard K8s NetworkPolicies, SecurityPolicy CRD adds vmSelector to select VMs, it also introduces some rule syntax including Drop and Reject actions, priority, rule level selectors(appliedTo).

An example of SecurityPolicy CR:

apiVersion: nsx.vmware.com/v1alpha1
kind: SecurityPolicy
metadata:
  name: db-isolation
  namespace: prod-ns
spec:
  priority: 1
  appliedTo:
    - vmSelector:
        matchLabels:
          role: db
  rules:
    - direction: in
      action: allow
      sources:
        - namespaceSelector:
            matchLabels:
              role: control
        - podSelector:
            matchLabels:
              role: frontend
      ports:
        - protocol: TCP
          port: 8000
    - direction: out
      action: allow
      destinations:
        - podSelector:
            matchLabels:
              role: dns
      ports:
        - protocol: UDP
          port: 53
      appliedTo:
        - vmSelector:
            matchLabels:
              user: internal
    - direction: in
      action: drop
    - direction: out
      action: drop
status:
  conditions:
    - type: "Ready"
      status: True
      reason: "SuccessfulRealized"

The example CR defines security rules for VMs which match the label "role: db" in Namespace prod-ns. The first rule allows traffic from Namespaces matching label "role: control" or Pods matching label "role: frontend" in Namespace prod-ns to access through TCP with port 8000. The second rule allows the selected VMs to access Pods with label "role: dns" through UDP with port 53. The third and forth rules are to drop any other ingress and egress traffic to/from the selected VMs.

Kubernetes NetworkPolicy style (from/to)

SecurityPolicy supports from and to fields as the preferred way to specify traffic peers, aligning with the standard Kubernetes NetworkPolicy syntax. The legacy sources and destinations fields are deprecated; use from and to instead. The following example is equivalent to the one above:

apiVersion: nsx.vmware.com/v1alpha1
kind: SecurityPolicy
metadata:
  name: db-isolation
  namespace: prod-ns
spec:
  priority: 1
  appliedTo:
    - vmSelector:
        matchLabels:
          role: db
  rules:
    - direction: in
      action: allow
      from:
        - namespaceSelector:
            matchLabels:
              role: control
        - podSelector:
            matchLabels:
              role: frontend
      ports:
        - protocol: TCP
          port: 8000
    - direction: out
      action: allow
      to:
        - podSelector:
            matchLabels:
              role: dns
      ports:
        - protocol: UDP
          port: 53
      appliedTo:
        - vmSelector:
            matchLabels:
              user: internal
    - direction: in
      action: drop
    - direction: out
      action: drop

Below are explanations for the fields:

spec: defines all the configurations for a SecurityPolicy CR.

priority: defines the order of policy enforcement within a cluster, the range is from 0 to 1000. Lower value has higher priority between policies.

appliedTo: is a list of policy targets to apply rules. As the CRD is namespaced scope, vmSelector or podSelector will be selected from the Namespace where the CR is created. vmSelector and podSelector cannot be in one entry as it would not select any workload. We can also have appliedTo in each rule entry, but if there is policy level appliedTo, it will take precedence over rule level.

rules: is a list of policy rules. The relative priority is based on the rule order in the list, rules in the front have higher priority than rules in the end.

action: specifies the action to be applied on the rule, including 'Allow', 'Drop' and 'Reject'.

direction: is the direction of the rule, including 'In' or 'Ingress', 'Out' or 'Egress'.

ports: define protocol, specific port or port range. ports.port will be treated as destination port. More details refer to section Targeting a range of Ports

from and to: defines a list of peers where the traffic is from/to. It could be podSelector, vmSelector, namespaceSelector and ipBlocks. podSelector and namespaceSelector in the same entry select particular Pods within particular Namespaces. vmSelector and namespaceSelector in the same entry select particular VMs within particular Namespaces. More details refer to section Behavior of from and to selectors

sources and destinations (deprecated): legacy names for from and to. These fields are still accepted for backward compatibility but users should migrate to from/to. If both sources and from (or destinations and to) are set in the same rule, from/to takes precedence.

status: shows CR realization state. If there is any error during realization, nsx-operator will also update status with error message.

Behavior of from and to selectors

There are 6 kinds of selectors that can be specified in an ingress from (or legacy sources) section or egress to (or legacy destinations) section:

podSelector: This selects particular Pods in the same namespace as the SecurityPolicy as ingress sources or egress destinations.

namespaceSelector: This selects particular namespaces for which all Pods and VMs as ingress sources or egress destinations.

namespaceSelector and podSelector: A single from/to entry that specifies both namespaceSelector and podSelector selects particular Pods within particular namespaces. Be careful to use correct YAML syntax; this policy:

  ...
  rules:
    - direction: in
      action: allow
      from:
        - namespaceSelector:
            matchLabels:
              user: alice
          podSelector:
            matchLabels:
              role: client
  ...

contains a single from element allowing connections from Pods with the label role=client in namespaces with the label user=alice. But this policy:

  ...
  rules:
    - direction: in
      action: allow
      from:
        - namespaceSelector:
            matchLabels:
              user: alice
        - podSelector:
            matchLabels:
              role: client
  ...

contains two elements in the from array, and allows connections from Pods in the current Namespace with the label role=client, or from any Pod in the namespaces with the label user=alice.

vmSelector: This selects particular VirtualMachines in the same namespace as the SecurityPolicy as ingress sources or egress destinations. E.g.

  ...
  rules:
    - direction: in
      action: allow
      from:
        - vmSelector:
            matchLabels:
              role: client
  ...

allows connections from VirtualMachines with the label role=client in the current namespace.

namespaceSelector and vmSelector: A single from/to entry that specifies both namespaceSelector and vmSelector selects particular VirtualMachines within particular namespaces. E.g.

  ...
  rules:
    - direction: in
      action: allow
      from:
        - namespaceSelector:
            matchLabels:
              user: alice
          vmSelector:
            matchLabels:
              role: client
  ...

contains a single from element allowing connections from VirtualMachines with the label role=client in namespaces with the label user=alice.

ipBlocks: This selects particular IP CIDR ranges to allow as ingress sources or egress destinations. Both IPv4 and IPv6 CIDRs are supported. E.g.

...
  rules:
    - direction: ingress
      action: allow
      from:
        - ipBlocks:
            - cidr: 192.168.0.0/24
...

Particularly, it can be used for single IP by suffix /32 (IPv4) or /128 (IPv6). E.g.

...
  rules:
    - direction: ingress
      action: allow
      from:
        - ipBlocks:
            - cidr: 100.64.232.1/32
...

IPv6 CIDRs are also supported in ipBlocks:

...
  rules:
    - direction: ingress
      action: allow
      sources:
        - ipBlocks:
            - cidr: 2001:db8::/32
    - direction: out
      action: allow
      destinations:
        - ipBlocks:
            - cidr: fd00::/64
...

Dual-stack configurations mixing both IPv4 and IPv6 ipBlocks in the same policy are supported as well:

...
  rules:
    - direction: ingress
      action: allow
      sources:
        - ipBlocks:
            - cidr: 192.168.0.0/16
        - ipBlocks:
            - cidr: 2001:db8::/32
...

Targeting a range of Ports

When writing a SecurityPolicy, you can target a range of ports instead of a single port. E.g.

...
  rules:
    - direction: in
      action: allow
      from:
        - podSelector:
            matchLabels:
              role: ui
      ports:
        - protocol: TCP
          port: 22
          endPort: 100
...

allows the Pods with label role=ui in the current namespace to the target port between the range 22 and 100 over TCP.

Policy priority and rule priority

The spec.priority in SecurityPolicy defines the order of policy enforcement within a cluster. If different SecurityPolicies have the same priority, in NSX side, it's not deterministic which policy will work at first, so we don't suggest the customer set the same priority for different SecurityPolicies.

In the same policy, the higher rule has the higher priority. E.g. in the policy:

...
  rules:
    - direction: in
      action: allow
      from:
        - podSelector:
            matchLabels:
              role: client
    - direction: in
      action: drop
      from:
        - podSelector: {}
    - direction: out
      action: drop
      to:
        - podSelector: {}
...

There're 3 rules in the array, rule[0] allows connections from Pods with the label role=client, rule[1] drops ingress connections for all Pods, rule[2] drops egress connections for all Pods. The traffic matching order is: rule[0] > rule[1] > rule[2], for a connection from Pods with the label role=client, it will be allowed and won't be dropped because the rule[0] will work.

Note

There are certain limitations for generating SecurityPolicy CR NSGroup Criteria, including: policy 'appliedTo' group, sources group, destinations group and rule level 'appliedTo' group. Limitations of SecurityPolicy CR:

  1. NSX-T version >= 3.2.0 (for mixed criterion)
  2. Max criteria in one NSGroup: 5
  3. Max conditions with the mixed member type in single criterion: 15
  4. Total of 35 conditions in one NSGroup criteria.
  5. Operator 'NotIn' in matchExpressions for namespaceSelector is not supported, since its member type is segment
  6. In one NSGroup group, supports only one 'In' with at most of five values in MatchExpressions, given NSX-T does not support 'In' in NSGroup condition, so we use a workaround to support 'In' with limited counts.
  7. Max IP elements in one security policy: 4000
  8. Priority range of SecurityPolicy CR is [0, 1000].
  9. Support named port for Pod, but not for VM.

IPv6 Support

Both SecurityPolicy CRD and standard Kubernetes NetworkPolicy support IPv6 CIDR blocks in ipBlocks. The except field in NetworkPolicy IPBlock also works correctly for IPv6 CIDRs. Dual-stack configurations (mixing IPv4 and IPv6 ipBlocks in the same policy) are supported.

For NetworkPolicy with IPv6 except clauses, the operator computes IP range exclusions and translates them to NSX-T IPAddressExpression entries using the IP range format (e.g., 2001:db8::b-2001:db8::ffff).