Thursday 1 September 2016

Complex data transformations with nested Heat intrinsic functions

Disclaimer, what follows is either pretty neat, or pure-evil depending your your viewpoint ;)  But it's based on a real use-case and it works, so I'm posting this to document the approach, why it's needed, and hopefully stimulate some discussion around optimizations leading to a improved/simplified implementation in the future.




The requirement

In TripleO we have a requirement enable composition of different services onto different roles (groups of physical nodes), we need input data to configure the services which combines knowledge of the enabled services, which nodes/role they're running on, and which overlay network each service is bound to.

To do this, we need to input several pieces of data:

1. A list of the OpenStack services enabled for a particular deployment, expressed as a heat parameter it looks something like this:


  EnabledServices:
    type: comma_delimited_list
    default:
      - heat_api

      - heat_engine
      - nova_api
      - neutron_api
      - glance_api
      - ceph_mon


2. A mapping of service names to one of several isolated overlay networks, such as "internal_api" "external" or "storage" etc:


  ServiceNetMap:
    type: json
    default:
      heat_api_network: internal_api
      nova_api_network: internal_api
      neutron_api_network: internal_api
      glance_api_network: storage
      ceph_mon_network: storage


3. A mapping of the network names to the actual IP address (either a single VIP pointing to a loadbalancer, or a list of the IPs bound to that network for all nodes running the service):

  NetIpMap:
    type: json
    default:
      internal_api: 192.168.1.12
      storage: 192.168.1.13


The implementation, step by step



Dynamically generate an initial mapping for all enabled services

Here we can use a nice pattern which combines the heat repeat function with map_merge:

  map_merge:
    repeat:
      template:
        SERVICE_ip: SERVICE_network
      for_each:
         SERVICE: {get_param: EnabledServices}




Step1: repeat dynamically generates lists (including lists of maps as in this case), so we use it to generate a list of maps for every service in the EnabledServices list with a placeholder for the network, e.g:

  - heat_api_ip: heat_api_network
  - heat_engine_ip: heat_engine_network
  - nova_api_ip: nova_api_network
  - neutron_api_ip: neutron_api_network
  - glance_api_ip: glance_api_network
  - ceph_mon_ip: ceph_mon_network

Step2: map_merge combines this list of maps with only one key to one big map for all EnabledServices




  heat_api_ip: heat_api_network
  heat_engine_ip: heat_engine_network 
  nova_api_ip: nova_api_network
  neutron_api_ip: neutron_api_network
  glance_api_ip: glance_api_network
  ceph_mon_ip: ceph_mon_network

Substitute placeholder for the actual network/IP


We approach this in two passes, with two nested map_replace calls (a new function I wrote for newton Heat which can do key/value substitutions on any mapping):



  map_replace:
    - map_replace:

    - heat_api_ip: heat_api_network
      heat_engine_ip: heat_engine_network 
      nova_api_ip: nova_api_network
      neutron_api_ip: neutron_api_network
      glance_api_ip: glance_api_network
      ceph_mon_ip: ceph_mon_network
         - values: {get_param: ServiceNetMap}
     - values: {get_param: NetIpMap}


Step3: The inner map_replace substitutes the placeholder into the actual network provided in the ServiceNetMap mapping, which gives e.g

  heat_api_ip: internal_api
  heat_engine_ip: heat_engine_network
  nova_api_ip: internal_api
  neutron_api_ip: internal_api
  glance_api_ip: storage
  ceph_mon_ip: storage

  
Note that if there's no network assigned in ServiceNetMap for the service, no replacement will occur, so the value will remain e.g heat_engine_network, more on this later..

Step4: the outer map_replace substitutes the network name, e.g internal_api, with the actual VIP for that network provided by the ServiceNetMap mapping, which gives the final mapping of:


  heat_api_ip: 192.168.1.12
  heat_engine_ip: heat_engine_network 
  nova_api_ip: 192.168.1.12
  neutron_api_ip: 192.168.1.12
  glance_api_ip: 192.168.1.13
  ceph_mon_ip: 192.168.1.13

Filter any values we don't want

As you can see we got a value we don't want - heat_engine is like many non-api services in that it's not bound to any network, it only talks to rabbitmq, so we don't have any entry in ServiceNetMap for it.

We can therefore remove any entries which remain in the mapping using the yaql heat function, which is an interface to run yaql queries inside a heat template. 

It has to be said yaql is very powerful, but the docs are pretty sparse (but improving), so I tend to read the unit tests instead of the docs for usage examples.

  yaql:
    expression: dict($.data.map.items().where(isString($[1]) and not $[1].endsWith("_network")))
      data:
        map:


          heat_api_ip: 192.168.1.12
          heat_engine_ip: heat_engine_network 
          nova_api_ip: 192.168.1.12
          neutron_api_ip: 192.168.1.12
          glance_api_ip: 192.168.1.13
          ceph_mon_ip: 192.168.1.13


Step5: filter all map values where the value is a string, and the string ends with "_network" via yaql, which gives:

  heat_api_ip: 192.168.1.12

  nova_api_ip: 192.168.1.12
  neutron_api_ip: 192.168.1.12
  glance_api_ip: 192.168.1.13
  ceph_mon_ip: 192.168.1.13


So, that's it - we now transformed two input maps and a list into a dynamically generated mapping based on the list items! :)


Implementation, completed


Pulling all of the above together, here's a full example (you'll need a newton Heat environment to run this), it combines all steps described above into one big combination of nested intrinsic functions:
 
Edit - also available on github


heat_template_version: 2016-10-14

description: >
  Example of nested heat functions

parameters:
  NetIpMap:
    type: json
    default:
      internal_api: 192.168.1.12
      storage: 192.168.1.13

  EnabledServices:
    type: comma_delimited_list
    default:
      - heat_api
      - nova_api
      - neutron_api
      - glance_api
      - ceph_mon

  ServiceNetMap:
    type: json
    default:
      heat_api_network: internal_api
      nova_api_network: internal_api
      neutron_api_network: internal_api
      glance_api_network: storage
      ceph_mon_network: storage


outputs:
  service_ip_map:
    description: Mapping of service names to IP address for the assigned network
    value:
      yaql:
        expression: dict($.data.map.items().where(isString($[1]) and not $[1].endsWith("_network")))
        data:
          map:
            map_replace:
              - map_replace:
                  - map_merge:
                      repeat:
                        template:
                          SERVICE_ip: SERVICE_network
                        for_each:
                          SERVICE: {get_param: EnabledServices}
                  - values: {get_param: ServiceNetMap}
              - values: {get_param: NetIpMap}


 

 

12 comments:

  1. Hi Steve,

    Thanks for the information.

    Could you help me with below query?


    I have a parameter ip_addresses which is of type comma_delimited_list
    I want this parameter to be list only, at the same time, I want to save this value to a file in a VM using cloud-init

    my-cloud-init:
    type: OS::Heat::CloudConfig
    properties:
    cloud_config:
    ssh_pwauth: True
    disable_root: false
    chpasswd:
    list: |
    cloud-user:user1
    expire: false
    hostname: host1
    write_files:
    - path: /tmp/ip.config
    owner: root:root
    permissions: '0777'
    content:
    str_replace:
    template: |
    ips=__IP_ADRS__
    params:
    __IP_ADRS__: {get_param: ip_addresses}

    Currently it throws error saying that property can only be integer or string in str_replace.
    So is there a way I can convert a list to string in Heat template.
    Is there a way that I can save the values in a list to a file in a VM?

    ReplyDelete
  2. Useful article, thank you for sharing the article!!!

    Website: blogcothebanchuabiet.com chia sẻ những câu nói mỉa mai người khác hay stt một mình vẫn ổn và giải thích hiện tượng chim sẻ bay vào nhà là điềm gì.

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. very useful information, the post shared was very nice.
    DevOps and Cloud Course Videos

    ReplyDelete
  5. Nice and excellent post. Very well explained. Thanks for the information....
    DevOps Online Training in Hyderabad
    DevOps Training Online

    ReplyDelete
  6. Nice Post and informative data. Thank you so much for sharing this good post, it was so nice to read and useful to improve my knowledge as updated one, keep blogging.
    Open Stack Training in Electronic City

    ReplyDelete
  7. Know about Quickbooks error 1328, how this error occurs, what are the causes and effects of this error and how you can rectify it here - Quickbooks error 1328

    ReplyDelete
  8. I happily found this website eventually. Informative and inoperative, Thanks for the post and effort! Please keep sharing more such blogs.

    QuickBooks Error 3371 Status Code 11118

    ReplyDelete