Salt autorun, a cautionary tale


if you have running salt for some time, most likely you have something like this

startup_states: 'highstate'

schedule:
  highstate:
      function: state.highstate
          minutes: 60

In your minion's config file. If you don't have it or don't know what, it is here is the explanation:

# Run states when the minion daemon starts. To enable, set startup_states to:
# 'highstate' -- Execute state.highstate
# 'sls' -- Read in the sls_list option and execute the named sls files
# 'top' -- Read top_file option and execute based on that file on the Master
startup_states: 'highstate'

from minion sample config file, /etc/salt/minion

schedule: will make will run the function defined (function:) on the schedule defined (minutes:)

more info at: http://docs.saltstack.com/en/latest/topics/jobs/schedule.html

This is very convenient, is an easy way to continuously ship updates to your infrastructure. Very cool indeed.

Then way is this post called a cautionary tale? Well, it may not work.

Some of my minions, less than 10%, where not running anything, they left in a state like this

$ ps -ef |grep salt
root     22766     1  0 Jan12 ?        00:30:12 /usr/bin/python /usr/bin/salt-minion
root     37401     1  1 Feb09 ?        05:56:34 /usr/bin/python /usr/bin/salt-minion

$ sudo salt-call state.highstate Test=True
local:
    Data failed to compile:
    ----------
        The function "state.highstate" is running as PID 37401 and was started at 2015, Feb 09 16:37:11.507736 with jid 20150209163711507736

$ tail /var/log/salt/minion
2015-03-02 06:37:20,324 [salt.utils.schedule][INFO    ] Running scheduled job: highstate
2015-03-02 06:37:20,336 [salt.utils.schedule][INFO    ] Running scheduled job: __mine_interval
2015-03-02 07:37:20,015 [salt.utils.schedule][INFO    ] Running scheduled job: highstate
2015-03-02 07:37:20,028 [salt.utils.schedule][INFO    ] Running scheduled job: __mine_interval
2015-03-02 08:37:20,783 [salt.utils.schedule][INFO    ] Running scheduled job: highstate
2015-03-02 08:37:20,799 [salt.utils.schedule][INFO    ] Running scheduled job: __mine_interval
2015-03-02 09:37:20,552 [salt.utils.schedule][INFO    ] Running scheduled job: highstate
2015-03-02 09:37:20,567 [salt.utils.schedule][INFO    ] Running scheduled job: __mine_interval
2015-03-02 10:37:20,340 [salt.utils.schedule][INFO    ] Running scheduled job: highstate
2015-03-02 10:37:20,353 [salt.utils.schedule][INFO    ] Running scheduled job: __mine_interval

The job 20150209163711507736 appears just to be hung without more information on the log

2015-02-09 16:37:14,855 [salt.loaded.int.module.saltutil][INFO    ] Loading cache from salt://_utils, for region2)
2015-02-09 16:37:14,855 [salt.fileclient  ][INFO    ] Caching directory '_utils/' for environment 'region2'
2015-02-09 16:37:14,889 [salt.loaded.int.module.saltutil][INFO    ] Syncing utils for environment 'base'
2015-02-09 16:37:14,890 [salt.loaded.int.module.saltutil][INFO    ] Loading cache from salt://_utils, for base)
2015-02-09 16:37:14,890 [salt.fileclient  ][INFO    ] Caching directory '_utils/' for environment 'base'
2015-02-09 16:37:14,957 [salt.state       ][INFO    ] Loading fresh modules for state activity
2015-02-09 17:37:11,764 [salt.utils.schedule][INFO    ] Running scheduled job: highstate

To fix it, you have to kill both salt-minion processes, and start salt-minion again.

Fixes are being discussed on github #16432

UPDATE:

While fixing this mess I found another bug in salt #21213