3
� �f1� � 0 @ s� d dl mZmZ d dlmZmZ d dlmZmZmZ d dl m
Z
m Z d dl m
Z
d dlmZmZmZmZmZ d dlmZ d dlmZ d d lmZ d d
lmZ d d lm Z d d l m Z d d
l m Z y d dl!Z!W n d dl"Z!Y nX d dl#Z#d dl$Z$d dl%Z%d dl&Z&d dl'Z'd dl(Z(d dl)Z)d dl*Z*d dl+Z+d dl,Z+d dl-Z-d dl.Z.d dl/Z/d dl0Z0d dl1Z1d dl Z d dl2Z2d dl3Z3d dl4Z4d dl5Z5d dl6Z6d dl7m8Z8 y d dl9Z9W n G dd� d�Z:e:� Z9Y nX y d dl;Z<W n d dl<Z<Y nX y d dl=Z=W n> e>k
�r0 y d dl?Z=W n e>k
�r* dZ=Y nX Y nX yd dl@Z@e@jAZBW n d dlCZCeCjDZBY nX deEe1�k�r|dd� ZFeFe1_Gy d dlHZHW n d dlIjJZHd dlKjLZMY nX G dd� deN�ZOdS )� )�execute_command�which)�datetime� timedelta)� Inspector�get_fqdn�get_server_name)� NoAgentSectionHeaderException�NoManifestFileException)�ProcessManager)�basename�exists�isdir�isfile�join)�PickleDatabase)�
PluginManager)�pprint)�ProgressPrinter)� ResultQueue)�Schedule)�exitN)�PluginBlacklisterc @ s e Zd ZdZdS )�SixTN)�__name__�
__module__� __qualname__�PY2� r r �"/usr/lib/fm-agent/library/agent.pyr - s r � check_outputc O s> d|krt d��tj| dtji|��}|j� \}}|j� }|S )N�stdoutz3stdout argument not allowed, it will be overridden.)�
ValueError�
subprocess�Popen�PIPE� communicate�poll)� popenargs�kwargs�process�outputZ
unused_err�retcoder r r �fJ s r- c @ s� e Zd ZdZdZdZdbdd�Zd d
� Zd d � Zdcdd�Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� Zddd d �Zd d � Zd!d"� Zd#d$� Zd%d&� Zd'd(� Zd)d*� Zd+d,� Zd-d.� Zd/d0� Zd1d2� Zd3d4� Zd5d6� Z d7d8� Z d9d:� Z ded<d=�Z d>d?� Z d@dA� Z!dBdC� Z"dDdE� Z#dFdG� Z$dHdI� Z%dJdK� Z&dLdM� Z'dNdO� Z(dPdQ� Z)dRdS� Z*dTdU� Z+dVdW� Z,dXdY� Z-dZd[� Z.d\d]� Z/d^d_� Z0dfd`da�Z1d;S )g�AgentZcustom�INFO� � �
� c C sN t jd�}tjj| j�s(| jjd� dS y0t| j�j � j
� j d�\}}t |�}t |�}W n | jj
dtj� � dS ytj|� W n tk
r� dS X yhddl}|jtjd| �j�d }td | �d
j� } || jkp�d | kp�d | j | k�r| j� dS W n Y nX |�rNtj� | d
|k�rN| jj
d| � tj |d� dS t j � }
| j |
�\} } | j �r�| jdk�r�| jj
d| j � t!� }
|
j"tj#d��}|
j$|| j�}| jj
ddj%|� � x |D ]}tj t |�d� �q�W dS |dk�r6| j&d7 _&|d
8 }| jj
d|||f � tj'|� | j(||d�S | jj
d| � dS dS )z/Check to see if it's safe to start up the agentZlock_mgrz-No existing PID file found, proceeding to runT�:z#Error reading existing PID file: %sr Nz/proc/%dz
ps -o cmd= %d� �pythonz%s_agentg N@z&Found stale agent process %s - killing� �rootz7Uninstalling. Killing all process from the username %s z/proczFound pids %s � r2 z]Found existing agent process %s, sleeping for %s and checking %s more times if safe to start.)�counterzAFound existing agent process %s, exiting to wait for it to finishF))�logging� getLogger�os�pathr
�pid_file�log�debug�open�read�strip�split�int�critical� traceback�
format_exc�getpgid�OSError�pwd�getpwuid�stat�st_uidr �lower�user�brand�remove_pid_file�time�kill�optparse� OptionParser�parse_arguments� uninstallr Zfilter_non_pid_process�listdirZget_process_from_userr �current_agent_delay�sleep�safe_to_start_agent)�self�timeoutZ
sleep_timer: r@ �pid� timestamprL ZusernameZpsout�parser�options�argsZmanagerZpidsr r r r] ` s^
"
zAgent.safe_to_start_agentc C s6 t j� }ttj� �}t| jd�}|jd||f � dS )z+Create a new PID file to track our instance�wz%s:%sN)r= �getpidrF rT rB r? �write)r^ r` �nowr- r r r �write_pid_file� s zAgent.write_pid_filec C s t jj| j�rt j| j� dS )z1Remove an old PID file to clean up on the way outN)r= r> r
r? �remove)r^ r r r rS � s zAgent.remove_pid_fileFc C s� d}t d| j |d�}|j|�}|r.|j� n
|jd� t d| j |d�}tjd| j| j| jf � |j� d}t d| j dd � |j� t
d
| j � d S ) Nr5 zNotifying %s of uninstall)�indentzERROR CONNECTINGzRemoving %r directoryzrm -rf %s %s %sz
Uninstalling %s
T)�sectionz
Uninstall of %s complete
) r rR Znotify_of_uninstall�finish�log_dirr= �system�db_dir�
config_dir�pkg_dir�print)r^ �aggregator_client�remove_instancerk �pp�successr r r rY � s
zAgent.uninstallc C s� t j� }y.|j| j�}|s"td��|jd�s4td��W n� t jtfk
r� | jj t
t j � �� t
jd dkr�tjdt| jd�j� �}n tjdt| jd�j� jd� �}|j|� Y n( tk
r� | jjt
t j � �� Y nX |S ) z�
Get the manifest configuration if it exists. Also, throw a deprecation
warning if the the manifest does not conform to the new-style format
(It must have an [agent] section heading).
zNo manifest file found�agentz�Using a manifest file without the section heading "[agent]" is deprecated; please add this heading to the file. Example:
[agent]
customer_key = customerkey
server_group = 123
r r3 z[agent]
�rzutf-8)� configparser� ConfigParserrC �
manifest_filer
� has_sectionr ZMissingSectionHeaderErrorr@ �warn�strrH rI �sys� version_info�StringIOrB �decodeZreadfp�info)r^ �manifestr| Zamended_manifest_filer r r � get_manifest� s&
zAgent.get_manifestc C sx t j� }d}tjj| j�rb| jjd� t| jd�}|j � }|j
� t j� }|j | j� | j ||�}| j ||�}|j
d�s�|jd� |jdd�s�|jdd| j� |jdd| j� d|jd�kr�|jdd�}ng }| jj|�}|r�|jdd|� tj� }|j
d� �r�|�r�|jdd�} yttj| �}
|
j�r0|
j} n|
j} tj | ��s�|jd� x8dD ]0} |j| d�}
|
dk �rR|jd| |
j d �� �qRW W n2 t j � d }t |�}| jj!d
j"|�� Y nX t| jd�}|j#|� |j
� tj$d| j � |dk �rdt%j&� }|j'|t| jd�j � �}t(|�}dd� |D �}t)|��rV| jjd� | jj*ddj+|�� n | jjd� n| jjd| j� |S )zo
Create/update the config file with the settings from the manifest.
Return the config.
NzExisting config file found�rbrx �aggregator_url�versionZplugin_blacklist� agent_proxy�https�http�/r5 zInstall proxy error: {}re z chmod 640 %sry c S s$ g | ] }|j d �s |j d�r|�qS )z+ z- )�
startswith)�.0�liner r r �
<listcomp>. s z&Agent.write_config.<locals>.<listcomp>zConfig file overwrittenzConfig diff:
%s� zNo change to config filezCreated new config file: %s)r� r� ),rz r{ r= r> r
� config_filer@ r� rB � readlines�closerC �copy_config_settingsr} � add_section�
has_option�set�agg_urlr� rc �get� _blacklisterZ update_list�
urlrequestZ
getproxies�urlparse�scheme�hostnameZ proxy_bypassrD r� �exc_infor �error�formatrg ro �difflibZDifferZcompare�list�lenrA r )r^ r� Z
new_configZold_config_linesZold_config_fileZ
old_configZoriginal_pluginsZupdated_pluginsZproxiesr� Zagg_url_optionZ agg_hostname�keyZp_url�errr� Znew_config_fileZdifferZ
diff_linesZchangesr r r � write_config� sn
zAgent.write_configc C sP xJ|j � D ]>}|j|�s"|j|� x$|j|�D ]\}}|j|||� q.W q
W |S )z�
Copy settings from the original to the destination, overwriting
destination's settings if they already exist.
)Zsectionsr} r� �itemsr� )r^ �originalZ destinationrl �option�valuer r r r� : s
zAgent.copy_config_settingsc C s� | j jd� | jr*td� | j jd� d S | j| j| j| jf}tj d| � | j jd| � | j
� }| j |�}i }|j d�r�|d }t
j|||||d�} y2|jdd�}td | � | j jd | � || _W n Y nX td
| j d d �}
td
d� |jd�D ��} |j d��r|d | _|j d��r>tdd� |jd�D ��} ni } y | j| j� | | �\}
}}}}W nd td� td� x*|jd�D ] }tdtjj|d� � �q�W | j jdt j � � t j � Y nX | �s�| �r�td| � | j jd| � t j � | j j d|
||||f � |�r&|| j!d< d|j"d�k�r�|jdd�j#� dk�r�d |j"d�k�r�x<|jdd �jd�D ]&}d t j$| j%|j&� f }tj |� �qpW |
�r�|
j'� n | j j(d |� |
j'd | � |
�r^|�r^|j)dd |� |j*t+| j,d!�� |�rtd"| j-| jf � | j jd#� nBt.d+t/|�d% �}d&| | }td'| j-| j|f � | j j0d(� n,td)| j-|| j-| j| j1f � | j j(d*|� | j2� d S ),NzBegining installationzAgent already installedzmkdir -p %s %s %s %sz Created directories: %s %s %s %sr� )� proxy_configrx r� z8Using manifest file aggregator for initial handshake: %sz
Handshaking with %s serversr5 )rk c s s | ]\}}||j d �fV qdS )z'"N)rD )r� r� r� r r r � <genexpr>q s z Agent.install.<locals>.<genexpr>Z
attributesc s s | ]\}}||fV qd S )Nr )r� r� r� r r r r� y s zI
There was an error in the initial handshake with the aggregator, pleasezIcheck your aggregator URL, and ensure you have connectivity to retrieve:
�,z %s
zv2/helloz Error in initial handshake: %szHandshake failed: %sz%s, %s, %s, %s, %s� log_level�enable_countermeasures�true� countermeasures_remote_pluginsz0%s %s/countermeasure.py install_plugins --url %szInstallation failed:
%szERROR CONNECTING: %s�
server_keyre zxInstallation of %s complete. Your server will now sync automatically with the %s ControlPanel.
z-Agent will automatically sync with aggregator�P r1 r9 z� Installation of %s complete. Please copy and paste the following
server key into the %s ControlPanel for your server: %sz^The server key must be manually entered into the Control Panel before agent will begin syncingz� Installation of %s had an error (%s). The %s is installed but it cannot sync correctly.
Please contact %s and send them the log file at %s zAggregator sync failed:
%sg D@)3r@ r� � is_installedrs rn rp rq �custom_plugin_dirr= ro r� r� r} �
aggregator�Clientr� r� r rR �dictr� r� Z handshake� get_all_ipsrE r> r r� rH rI r� r rA �dbrc rP �
executable�bin_dirrD rm rG r� rg rB r� rr rF r� r~ �log_file�migrate_config)r^ r� r� r� � customer_key�dirsr� �configr� rt rv Zagent_settingsZserver_attributesrw � found_serverr� r� �url�cmdZpaddingr r r �installG s�
z
Agent.installc C s g }t d�}|}dtjks$dtjkr,|d }dtjkr�t d�}|r�td| �\}}|dkr�x�|jd �D ]8}|j� jd
�sf| sf|d kr�qf|j� }|j|d � qfW n�|o�dtjk�rt|�\}}|dk�r6tjd kr�tj d|�}dd� |D �}ndd� tj d|�D �}n.t d�} td| �\}}dd� tj d|�D �}dd� |D �}dd� |D �}d|k�rjd||j
d�< |�s�| j j d� t
j| j| j| jd�}
y|
j� g}W n: tk
�r� } z | j jd j| �� g }W Y d d } ~ X nX |�s�| j jd � n| j jd |� |S ) NZifconfig�sunos�aixz -azhp-ux�netstatz%s -inr �
�namer� r3 �freebsd�darwinzinet6? (.+?)\sc S s g | ]}|j � jd �d �qS )�%r )rD rE )r� �ipr r r r� � s z%Agent.get_all_ips.<locals>.<listcomp>c S s g | ]}|j d ��qS )zaddr:)rD )r� �xr r r r� � s r� z %s addr showc S s g | ]}|�qS r r )r� r� r r r r� � s c S s g | ] }|r|�qS r r )r� r� r r r r� � s c S s( g | ] }|j � jd �d jd�d �qS )r� r r� )rD rE )r� r� r r r r� � s �1z::1z@Unable to retrieve IP address(es) locally, contacting aggregator)r� z IP address lookup failure: {}z"Unable to determine IP address(es)zIP addresses: %s)r� r� )r r� �platformr rE rP r� �append�re�findall�indexr@ �warningr� r� r� r� r� Z get_local_ip� Exceptionr� r� rA ) r^ ZipsZ
ifconfig_pathZ ifconfig_cmdr� �coder+ �lr� Z ip_addr_pathrt �er r r r� � sR
zAgent.get_all_ipsc C sP t jj|�si S y(t|�j� j� jd�}tdd� |D ��S td� i S d S )Nr� c S s" g | ]}t ttj|jd ����qS )�=)r� �mapr rD rE )r� r� r r r r� s z9Agent.get_old_style_config_properties.<locals>.<listcomp>zError reading manifest file) r= r> r
rB rC rD rE r� rs )r^ �manfileZmfr r r � get_old_style_config_properties� s z%Agent.get_old_style_config_properties�r+c C sz t ||�}d}xFtd�D ]:}y tj|tjtjB � d}P W q tjd� Y qX qW |rv| jj d| � |j
� d S |S )NTr2 Fg �?z Could not acquire lock on %s) rB �range�fcntl�flockZLOCK_EXZLOCK_NBrT r\ r@ � exceptionr� )r^ �fname�modeZofile�locked�ir r r �
_open_file s
zAgent._open_filec
C s0 t | j��r(| j| j�}|s i S i }y\tj|�}xL|D ]D\}}}tj|d�}t|�}||tj |j
� �g|d||j d�f < q6W W n | j j
d� i }Y nX t|j� �}|jdd� d� i }x<|D ]4\}}}||kr�||gg||< q�|| j||g� q�W |jd� |j� tj|tj� |j� |S i S d S ) Nz%Y-%m-%d %H:%M:%Sz%s:%sz
%Y%m%d%H%Mz"Unable to parse custom metric filec S s | d S )Nr1 r )�vr r r �<lambda>* s z)Agent.get_metric_values.<locals>.<lambda>)r� r )r
� report_filer� �csv�readerr �strptime�floatrT �mktime� timetuple�strftimer@ r� r� �values�sortr� �seek�truncater� r� �LOCK_UNr� )r^ �csvfileZ
unique_valuesZ
csv_reader�textkeyr� ra Z
custom_valuesr r r �get_metric_values s6
.
zAgent.get_metric_valuesc C s� t | j�r�| j| j�}|s i S tj|�}ytdd� |D ��}W n$ tk
rb | jjd� i }Y nX |j d� |j
� t j |t j
� |j� |S i S d S )Nc S s g | ]}|d |d f�qS )r r5 r )r� �rowr r r r� E s z0Agent.get_registered_metrics.<locals>.<listcomp>z)Error reading custom metric register filer )r
�
register_filer� r� r� r� r� r@ r� r� r� r� r� r� r� )r^ r� Z csvreader�metricsr r r �get_registered_metrics= s
z Agent.get_registered_metricsc C sJ g }x@t | jd j� �D ]*\}}d|j|jf }||kr|j|� qW |S )N� schedulesz%s.%s)r� r� r� �plugin_textkey�resource_textkeyr� )r^ Zexisting_tkeys�sr_id�schedule�tkeyr r r �get_existing_metricsV s zAgent.get_existing_metricsc C s |dkr|j d�rdS dS )Nzcom.pnp-hcl.dominostatszMem.PID.TF)r� )r^ r r r r r �
ignore_metric^ s
zAgent.ignore_metricc C s( ddg}ddddg}ddddg}| j � }| jjd � i }i }g }| jg} |jd
d �r`|jd
d �pbd }
|
rx| j|
jd �� | j} |jd
d
�r�|jd
d
�p�d } | r�t | �} g }
xr| D ]j}t
|�s�q�t |
�| kr�P | jjd|� x>t j
|�D ]0}t |
�| kr�P tt||��r�|
jt||�� q�W q�W �x�|
D �]�}t|�}t j|t j��sV| jjd|� �q&t|d�}�y0| jjd|� tj|j� �}|j� x2|D ]*}|t|j� �k�r�t jd||� �q��q�W |jdg �}�x�|D �]�}x8|D ]0}|t|j� �k�r�t jd|� t |� �qܐq�W | j |d |d ��r*�q�yJyt |d �}W n4 t k
�rp t j!|d d�}t"j#|j$� �}Y nX W n2 t k
�r� | jjd|d |d � �w�Y nX |d |f}d|d |d f }||k�r@||k�r�|| j%dg �j|� nJ|jdd �}|d k�r|d }|d |d |d ||jdd �|gd�||< n|j%|g �j|� �q�W |jdg �}�x |D �]} x8|D ]0}|t| j� �k�rxt jd |� t | � �qx�qxW yJyt | d �}W n4 t k
�r� t j!| d d�}t"j#|j$� �}Y nX W n2 t k
�r( | jjd | d | d � �wlY nX |d | d || d | d d �} d | k�r`| d | d < d | k�rv| d | d < |j| � �qlW t j&|� W n� t k
�r |j'�r�t||j(�}| jjd!|� |j)d"� | jj|j� � | jjt*j+� � | jjd#� |j� t j&|� �w&Y nX �q&W |||fS )$Nr Zplugin_category_namer� r� �unitra � description�actionz Processing incoming import filesrx Zmetric_incoming_directoryr� Zmax_incoming_files_overridez
Looking in %sz&Can not delete %s so will not process.zr+z
Processing %sz1Can not process file %s! Missing required key: %sr z0Can not process metric! Missing required key: %sz%Y-%m-%d %H:%M:%Sz,Could not process timestamp %s for metric %sz%s.%s� first_values�label)r Z plugin_namer r r
r
� incidentsz2Can not process incident! Missing required key: %sz.Could not process timestamp %s for incident %s)r r ra r r Z match_key�metadatazError processing %s:r z
Deleting file),r r@ r� �custom_import_dirr� r� �extendrE �MAX_IMPORT_FILESrF r r� r= rZ r r r� r �access�W_OKr� rB �json�loadsrC r� r� �keysr; r r r� r r� �calendarZtimegmr� �
setdefaultrj �closedr� r� rH rI ) r^ r� Z req_top_keysZreq_metric_keysZreq_incident_keysZexisting_metricsZ new_metricsZ
new_values�custom_incidentsZ import_dirsZadditional_dirsZ max_filesZ max_override�files� directoryr- � full_pathr� �jZreqr �mZunix_timestampra Z new_valuer r r Zincident�objr r r �process_importsd s�
zAgent.process_importsc C s� dt � i}tjj| j�r�| j| j�}|s,|S | j| j�}|jd� |j� t j
|t j � |j � ytj
| j� W n Y nX |S | jr�t� }|r�||d< |S d S )NZfqdnr � server_name)r r= r> r
�update_config_filer� r� r� r� r� r� r� r� rj �is_fortisase_installr )r^ r� r� Z
propertiesr$ r r r �get_update_config� s(
zAgent.get_update_configc s� |� _ |� _|� _|� _|� _|� _|� _tj� � _ d� _
t j � dkpPt j
� dk� _|
� _t jj| � j�� _t� jd� j �� _t jj| � j�� _t� jd� j �� _t jj| � j�� _t� jd� j �� _t jj|
� j�� _t jj� jd�� _t jj� jd�� _t jj| d� j �� _t jj| � j�}t jj|d �� _ t jj| � jd
�� _ d t j j � k�r~t jj|d �� _!t jj|d
�� _"n^dt j j � k�r�t jj� jd �� _!t jj� jd
�� _"n(t jj| � jd �� _!t jj| � jd
�� _"d� _#d� _$d� _%d� _&d� _'d� _(d � _)d� _*d � _+�yjt,� j��s*t-d��t jj,� j��sJt-dj.� j���t/j0� }|j1� j� |j2d��rt|d � _+dt j k�rb|j3dd��r�|j4dd�j � dk� _$|j3dd��r�|j4dd�� _%|j3dd ��r�|j4dd �� _&|j3dd ��r�|j4dd �� _'|j3dd ��r |j4dd �j � dk� _(|j3dd ��r8|j4dd �� _)|j3dd ��rbd!|j4dd �j � k�rbd� _*|j4dd"�}|�s|t-d#��W n< t-k
�r� } z t j5j6d$j.|�� d� _#W Y d d }~X nX yt7|j4dd%��}W n d&}Y nX d't j8� k�r�d(t _ d� _9� j:� t;j<� j=j>�� _?yt@|j4dd)��� _AW n t-k
�rF |� _AY nX � jB||d*��s|� fd+d,�� _C� j?jDd-� t jEd.� � jF� � jG� � _H� jI� tJ� � _K� j?jLd/� d S )0Ni r z%s.dbz%s.logz %s_agent.cfg�countermeasuresZincomingz%s-agent-manifestz agent.pidz
update-configr� �registerZreportr� TFZ
demserviceZ
updateservicez /tmp/com.fortinet.fortimonitorzNo bin directoryzNo config file {}r� ZdemZenabledr� Z server_portrx zupdateservice.portZipc_path� auto_update�scheduled_updateZhandshake_typeZ forticlientr� zMissing server keyzInitialize exception: {}� safe_counterr3 ZVMkernelZvmwareZstartup_timeout)r: c s � j jd�S )NzPreventing pid file removal)r@ r� r )r^ r r r� y s z Agent.__init__.<locals>.<lambda>z=Exiting without running - other agent process already runningr5 zActivity started)MrR r� r� rQ �lib_dirr� rr �tempfileZ
gettempdir�tmp_dir�metadata_rebuild_freqr= �getuid�geteuid�is_root�acceptable_sync_delayr> r rp �db_filern r� rq r� r� �!countermeasures_custom_plugin_dirr r| r? r% r� r� rP r� r� r� �has_dem�dem_port�update_service_port�ipcPathr* r+ r& r� r
r� r� rz r{ rC r} r� r� �stderrrg rF �unamer[ �set_up_loggingr; r< � __class__r r@ r� r_ r] �__del__r� r ri �open_dbr� r� r r� r� )r^ rR r� r� rQ r� r- rr r_ Zbase_config_dirZbase_custom_plugin_dirZ
base_data_dirZ base_log_dirr4 Zdata_dirr� r� r� r, r )r^ r �__init__ s�
zAgent.__init__c C s� | j d r| jjd� dS tj� }|j| j�r�d}|jd�sV|jd� d}| jjd� |jd�r�x0|j d�D ]"\}}|d kr�d
}|j
d||� qlW |j d� d}| jjd � |r�t | jd �}|j
|� |j� d| j d< dS )
z�
Update agent configs to use "[agent]" instead of "[AgentConfig]" as
the main heading and "aggregator_url" instead of "agg_url" (in order to
match the option in the manifest file).
�config_migratedz Config is in the correct formatNFrx Tz Added [agent] section to configZ AgentConfigr� r� zMCopied deprecated [AgentConfig] section to [agent] and removed it from configre )r� r@ r� rz r{ rC r� r} r� r� r� Zremove_sectionrB rg r� )r^ r� Zconfig_has_changedr� r� r� r r r r� � s.
zAgent.migrate_configc C s | j � d S )N)rS )r^ r r r r? � s z
Agent.__del__c C s t j� }tjj| j�s(tjdj| j�� yt| j d�}W n> t
k
rv t d| j t t
j� �f � t d� t j� }Y n X |j� t jj| j dd dd�}|jt jd �� |j|� |jt j� | j� }ytt |d
j� �}W n tt | j�}Y nX |j|� d S )
Nz mkdir -p {}�az Cannot open log file %s: "%s"zLogging to stderr instead� i r1 )ZmaxBytesZ backupCountzA%(process)d) %(asctime)s - %(name)s - %(levelname)s - %(message)sr� i i P ) r; r< r= r> r rn ro r� rB r� �IOErrorrs r rH rI Z
StreamHandlerr� ZhandlersZRotatingFileHandlerZ setFormatterZ FormatterZ
addHandler�setLevelZNOTSETr@ �getattr�upper�DEFAULT_LOG_LEVEL)r^ Z root_loggerr� �handlerr� r� r r r r= � s0
zAgent.set_up_loggingc C s | j r(|jdddd� |jddddd � |jd
dd dd � |jd dd
dd � |jddddd � |jdddd� |jddddd� |jddddd� |jddddd� |jdddd d � |jd ddd d � |jd ddd!d � |jd"ddd#d � |j� \}}||fS )$zJ
Return the options and arguments parsed from the parser.
z --server-keyr� Zstore)�destr z--rebuild-metadata�
store_true�rebuild_metadataF)r rK �defaultz--status�statusz--stats�statsz --from-cron� from_cronz --aggregatorr� )r rK z --installr� )r rN rK z --uninstallrY z--remove-instanceru z--customer-keyNr� )rN r rK z --unpause�unpausez--list-containers�list_containersz --rebuild-container-metadata�rebuild_container_metadata)r� Z
add_option�
parse_args)r^ rb rc rd r r r rX � s zAgent.parse_argumentsc � L C sJ t j� }| jr| j}nd}d }tj� }|j| j� | jr�|r�|g kr�y|jdd�}W n d }Y nX y|jdd�px| j | _ W n d | _ Y nX d| j
|| j f }n d| j
f }t j |d�}| j
|�\}}|j�rt|| jt| jd�| j�} tj| ||d | � |j�r$tj|d |d
|d � |j�rXtj| j | j|| jd �}
| j|
|j� t� | j �sj|j �r^|j�rz|j| _ |j �p�d } d
t j j � k�rF| j!d f| j"d f| jdf| j#df| j$dfg} x4| D ],\}
}t%j&dj'|
�� |�r�t%j(|
|� �q�W xL| j)| j*gD ]<}t+|d��}W d Q R X t%j,j-|��s| j.j/dj'|�� �qW | j | j | j|| � d S dddddddg}d}x"|D ]}t0||d ��rzd}P �qzW |�s�d| j1| j|| j f }t2|� | j.j3|� d S |j4�r�t2d� d |d< |j5� d S |j6�r�d |k�s|d i k�r t2d � d S |d }t2d � t2d � xV|j7� D ]J\}}|jd d!�}|jd"d!�}|jd#d!�}t2d$|||d d%� |f � �q>W d S |j8�r�d|d< t2d&� | j.j3d'� d S d}�y6d} |j9�r�d} |j9} t2d(| � |j:dd| � |j�r d} |j} t2d)| � |j:dd| � | �r<|j;t+| jd*�� td+� | j<�rf|j= �rf| j.j>d,� t2d,� d S |jdd�}tj| j | j|| jd �}
|�s�t2d-� td.� | j?�ryLd} | j@d/| jAd0�} | d k�s�d1| k�r�d} | �s�| jB|��r�| jC|
� W n Y nX t|| jt| jd�| j�} | jD� }!|!�r4| jE|!� d2|jFd�k�r8|jdd2�j � d3k�r8d4|jFd�k�r8d5|jFd�k�r8tG|jdd5��d6 }"d7|k�s�tHjH� |d7 |"k�r
xL|jdd4�jId8�D ]6}#| j.j3d9|# � d:t jJ| jK|#jL� f }$t%j&|$� �q�W tHjH� |d7< n.d7|k�r8| j.j3d;|d7 |" tHjH� d< � t j� }%g }&| jM� }'| jN|�\}(})}*d=|k�st|d= d k�r|i |d=< �x6tO|d j7� �D �] \}+},d>|,jP|,jQf }-d?}.|,jP| jRk�r�|,jQ|'k�r�|-|)k�r�|,jS|%tT|.d@� k�r�| j.j3dA|,� �q�tT|,jUd@�}/tT| jVd@�}0|%|/ |0 |,_S|-|)k�r�| jWj|-i �jdBdC�}1xJ|)|- D ]>\}2}3|2d k �rd|2|19 }2|2d k�rr�qJ|&jX|+|3|2f� i }4�qJW �q�|,jP| jRk�r|,jQ|'k�r��q�| jWj|,jPi �jdBdC�}1x�|'|,jQ D ]>\}2}3|2d k �r�|2|19 }2|2d k�r��q�|&jX|+|3|2f� i }4�q�W n�|,jP| jYk�r8| j.j3dD|-� �q�nxt j� }5tHjH� }6|,jZ| |d= j|,j[i ��\}2}4tHjH� }7| j.j\dE|,|2|7|6 f � |2d k�r��q�|&jX|+tHj]|5j^� �|2f� �q�W | j.j3dFt j� |% � |dG j_|&� |d � rFtHjH� |d k � r&|j5� |d tHjH� dH }8| j.j3dI|8 � d S d |d< |j5� | j.j3dJ� d S tHjH� }9d }i }:g };d }<t`jad+dK�d+k}=d}>dL|k� r�d+|dL< tHjH� |dL | jbk� r�d}>|j=� r�d}>dM|k� r�tHjH� |dM< t%j,jc| j�}?|?|dM k� r�d}>|?|dM< |>�rp| j.j3dN� tHjH� |dL< dO|jFd�k�
r*| jd|jddO�� | je� }@|@�
sR|j=�
sR|=�
sR|dP �
r�| jf}|@�
rl| j.j3dQ� n@|j=�
r�| j.j3dR� n*|=�
r�| j.j3dS� n|dP �
s�| j.j3dT� tg| �jh|!�}<| ji� }:g };d2|jFd�k�r |jdd2�j � d3k�r �xt%j,j| jdUd�| j#fD �]�}At%j,jj|A�� s � qt j,jX|A� �x�t%jk|A�D �]�}B|BjldV�� r�|BjmdW� � r�ytnjo|Bd d�� �}CW n | j.jpdY|B � � w:Y nX tq� r�trjs� r�tqt+t%j,j|A|B��j� �jt� }Dn$tqt+t%j,j|A|B��j� judZ��jt� }Dnd[}Dx�tO|Cjvj7� �D ]�\}E}Ft jwd+ dXk� r tx|F�txk� s@t jwd+ d\k� r�tx|F�tyjzk� r�|Ejld]�� r�y(|F� }G|;jX|Gj{|Gj||Gj}|D|Gj~d^�� W n Y nX � q�W |Bjld_�� r:yt+t%j,j|A|B��}HW n: | j.j>d`t%j,j|A|B� � | j.j>tj�� � � w:Y nX |Hj� }Itq�
rtq|IjudZ��jt� }Dnd[}D|Hj�� yt�j�|I�}JW n: t�k
�
r\ | j.j>da|B � | j.j>tj�� � � w:Y nX dbdcdddeg}K|Jj�� }Ld}Mx@|KD ]8} | |Lk�
s�|Jj| � �
r|| j.j>df| |Bf � d}MP �
q|W |M�
s q:|Jjdc�}N|;jX|N|Jjdb�|Jjde�|D|Jjdg�d^�� � q:W � qW dh|k�s|dh �rTt�j�� }O|Odik�r>d|dh< | j.j3dj� n|Odkk�rT| j.j3dl� dh|k�rp|dh �rpd|d< i }Pi }Qdm|k�r�|dm �r�t�|dm �|Qdm< dn|k�r�|dn �r�t�|dn �|Qdn< do|k�r�|do �r�t�|do �|Qdo< dp|k�r|dp �rt�|dp �|Qdp< g }Rdq|k�r |dq }R�y�g }S| j.j3drt�|&�t�|S�� |�rlt�dsdt� tO|j7� �D ��}T| j.j\du|T� d}U|d
d+k�s�|d i k�r�d}U|>�s�|dv �r�d}Ut`jad.| j��p�d.�}VtHj�|V� |dw �r tHjH� |dw k �r�|dG j�t�|d ��}Wnd |dw< |dG j�� }Wn |dG j�� }Wg }Xg }Yd%}Zdx|k�rd|dx }[x,t�t�t�|[�|Z��D ]}\|XjX|[j�d+�� �qJW dy|k�r�|dy }]x&t�t�|]��D ]}\|YjX|]j�d+�� �q�W | j�|�}^y�d+t _�tO|(j�� �}(|�rh|
j�|W|S||;|<|X|Y| j�� |(|*| j�� | j�� |R|U|Q| j?|^|:dz�}Pd |dm< d |dn< d |do< d |dp< g |dq< d |dv< |Pjd{i �|Pjd|i �|Pjd}g �|Pjd~i �d�}_| j�|_� n | j.j3d�� W nd |dG j_|W� x(|^j�� D ] }`||` }a|aj_|^|` � �q�W | j.jpd�� | j.j\d�|W� |j5� d S |Pjd�d��r�d|dP< |d
d.7 < t j� j�d��|d < W n | j.jpd�� Y n.X |>�rB| j�� |d�< | j.j3d�tHjH� |9 |V � d2|jFd�k�r�|jdd2�j � d3k�r��x|PjdUg �D �]}b|bjd��}D|bjd�g �}c|bjd�i �}dd[}ey>d�|D }ft%j,j| j�|f�}et+|ed��}g|gj;t�j�|d�� |gj�� W n: t�k
�r6 | j.j>d�|D|cf � | j.j>tj�� � Y nX | j.j3d�|D|cf � |c�r�d�t jJ| jK|Dd�j|c�f }$|e�r~|$d�|e 7 }$t%j�t%j�t jJ|$jI� � �q�W | j�|Pjd g �� |Pjd�g ��r�| j.j3d�t�tO|Pd� j�� ��� d|Pd� k�r|Pd� d }htHjH� |h |d< dw|Pd� k�r,|Pd� dw }htHjH� |h |dw< dm|Pd� k�rf|Pd� dm }it�j�d�|i| j�f dd��}j|j|dm< d�|Pd� k�r�|Pd� d� }k|k|dG _�d�|Pd� k�r�|Pd� d� }l|l|dG _�do|Pd� k�r8yb|Pd� jdo�}|jd��}m|md k�r�d�}mtG|m�}md�}nt�d���rd�|m|nf }nt�j�|ndd��}o|o|do< W n tj�� |do< Y nX dp|Pd� k�r y�|Pd� jdp�}|jd��}p|jd��}m|md k�rxd�}mtG|m�}m|pd k�r�t�j�| j �}q|qj�d k�r�t�j�d�| j �}q|qj�}pd�}r|r|p; }rt�d���r�d�|m|rf }rt�j�|rdd��}s|s|dp< W n tj�� |dp< Y nX |Pd� jd��}t|td k �r�|tj�� }ty,t0t�|t�}u|t|d�< | j.j�|u� t�j�}vd�}wW n t�k
�rx t�j�}vd�}wY nX | j.j.|v|w|t � dn|Pd� k�r�| j�|| j| j1�|dn< d�|Pd� k�r�d+|dL< d |d�< d�|Pd� k�r|d2|jFd�k�r||jdd2�j � d3k�r|d4|jFd�k�r|d5|jFd�k�r|xL|jdd4�jId8�D ]6}#| j.j3d9|# � d:t jJ| jK|#jL� f }$t%j&|$� �q6W tHjH� |d7< d|Pd� k�r�d|d< d�|Pd� k�r�d}dv|Pd� k�r�d|dv< d�|Pd� k�r�y| j�|� W n Y nX | j<�r| j.j3d�� t2d�� �n�dh|k�r�|dh �r�d |k�r*i |d < d}xd|k�rF|d �rFd}xd|d< |d }ytO|yj�� �}zyt�j�|| |y|xd��}{W nF t�k
�r� t j�� d d\� \}|}}| j.j>|}� | j.j>d�� d }{Y nX |{�r�d�d�� |{D �}~g }xd|{D ]\}�|�d� d d�� }�|�|y|�< |x�s
|�|zk�r|jX|�� d�|�k�r�|�d� �r�|�d�= |jX|�� �q�W g }Yx*|yj7� D ] \}�}�|�|~k�rL|YjX|�� �qLW x|YD ] }�|y|�= �qtW dx|k�r�g |dx< dy|k�r�g |dy< |dx j�|� |dy j�|Y� | j.j3d�t�|�� | j.j3d�t�|Y�� | j�|� W n | j.jpd�� Y nX | j�|||
|d�� |j5� | j.j3d�j't j� | j�� �� d S )�NFrx r� r� zF%%prog [options]
%s, server key: %s, aggregator endpoint: %s
zGsudo python %%prog --install [--customer-key=YOUR_CUSTOMER_KEY]
%s)�usage�pluginsr � num_syncs� last_sync)r� r� i� z mkdir -p {}za+zInstaller did not create {}rQ r� rM rR rS rT Tz5%s Agent v%s, server key: %s, aggregator endpoint: %sz.Unpausing agent, will run as usual on next run�pauseZdocker_containerszNo monitored containerszMonitored Containers:
z$CONTAINER ID IMAGE COMMAND STATUSZImage�?ZCommandZStatusz
%s %s "%s" %sr0 zMetadata queued for rebuildz!Container metadata rebuild queuedzSetting server key to %sz!Setting aggregator endpoint to %s�wbr z.Linux agent should not run if executed as rootz1No server key found, please re-install the agent.r5 zschedules-init)�port�nor� r� r� Z countermeasures_refresh_pluginsi Z countermeasures_last_refreshr� z*Refreshing CounterMeasures plugins from %sz2%s %s/countermeasure.py install_plugins --url %s &z+Waiting to refresh CM plugins in %d minutes�< � anomaliesz%s.%srD )�secondsz%r too early to check�scaleg �?z No custom value or plugin for %sz %r returned %s in %.2f secondszRunning all plugins took %s� result_queueg N@zPPause command recieved. Processing stopped. Process will resume in %.2f minutes.z=Pause duration exceeded, unpausing the agent for the next runi� Zlast_metadata_time�last_config_file_timezRebuilding plugin metadataZcustom_plugin_url� has_connected_with_aggregatorzmetadata changedzrebuilding metadataz!randomly forcing metadata rebuildz%we've never pushed up metadata beforer( z.py�__r3 zUnable to import module %szutf-8r� r1 ZCountermeasure)r� r� �author�hashr z.jsonzUnable to open %sz+%s file is not a valid json file to be readr� r� �commandrg z7%s is missing from the countermeasure declaration in %sr Zdocker_supportedrw zDocker supportedz
no-permissionz*Missing permission to access Docker socketZ log_request� diagnostics� socket_statsZmtr�auto_topo_scansz1Syncing with aggregator: %d results, %d anomaliesc s s* | ]"\}\}}|t t|j� ��fV qd S )N)r� r� r )r� Z
plugin_key�_Zplugin_metadatar r r r� � s z Agent.main.<locals>.<genexpr>zMetadata summary: %r�sync_schedules�
single_result�discovered_containers�deleted_containers)Z dem_enabledZdem_service_results�fortisase_attributes�icmp_server_resources�monitor_schedules� traceroutes�traceroute_checks)rs rt ru rv z"No server_key found, skipping syncz Could not sync with aggregatorzSaving results locally: %rr� z%m/%d/%Y %H:%Mz Error syncing with aggregator�
last_metadatazsyncing took %.2f secondsrh �textkeysr z countermeasure-metadata-%s.jsonre z1Failed parsing countermeasure metadata for %s: %sz#Queueing countermeasures for %s: %sz7%s %s/countermeasure.py execute --hash %s --textkeys %sr9 z --metadata-file %sZcommandszgot %d agent commandsz tail -%d %s)�shell�queue_batch_size�queue_max_resultsr_ r2 zss -t -u -r 2>&1z
timeout %d %s�hostzhttp://zmtr --csv -c 1 %s 2>&1r� zSet log level to "%s"z Invalid log level command: "%s"Zmetadata_resyncZrefresh_countermeasuresZ update_agentZget_logsz9Linux agent running as root, skipping container discovery)Zrebuildz�Docker has been enabled but the fm-agent user needs to be added to the docker group.
You can do so with `sudo usermod -a -G docker fm-agent`c S s g | ]}|d dd� �qS )�IdN� r )r� �cr r r r� � s z Agent.main.<locals>.<listcomp>r} r~ �updatedz$Discovered %d new/updated containersz!Found %d newly deleted containerszError in main loop)r� r� �
agg_client�forcezActivity finished in {}s���)�r rh r� r� rz ZRawConfigParserrC r� r� r� rr rV rW rX rO r r r- r� �displayrP rY r� r� r� r� ru r r� r� r� r� rP rp rq r6 r r= ro r� �chmodr� r� rB r> r r@ r� rG rR rs r� rR �saverS r� rT r� r� rg r3 rM r� r7 � _sendReceiver8 �_agent_version_updated�_init_dem_schedules�get_dem_wifi_infoZadd_dem_wifi_resultsrc rF rT rE r� r� rD r� r# r� r r �CUSTOMZnext_check_timer � frequencyr[ r� r� rW �check�idrA r� r� �update�randomZrandintr0 �getmtimeZinstall_remote_pluginsZis_metadata_staler r Z
get_all_facts�get_fortisase_attributesr
rZ �endswithr� � p_importlibZ
import_moduler� �sha_func�sixr Z hexdigest�encode�__dict__r� �type�typesZ ClassTyper� r� rg r rH rI r� r r r� r �container_discoveryZ check_accessr r� r� r4 r\ � pop_resultsr� �min�pop�_getDemResults�tracebacklimitr� �syncr r' r� �_updateDEMServiceSchedulesr� Zhashed_metadatar/ �dumps�spawnvp�P_NOWAIT�update_schedulesr# r r� rz r{ r r� r� rH r; rF r/ �AttributeErrorZWARNING�build_diagnostics� upload_logsZdiscover_docker_containersr� r �run_auto_topo_scans�checkForUpdate�
total_seconds)�r^ Z
activityStartr� r� r� rV rb rc rd rW rt r� Zdirs_to_create�dirZpermsZrfileZrfZ
valid_optionsZ option_givenZ valid_option�msgZ
containersZshort_idr Z
cont_imageZ cont_commandZ cont_statusZrequested_auto_updateZjust_set_option_and_quitr� ZaggZneeds_schedulesZschedules_receivedZ wifi_infoZ
refresh_cycler� r� Zall_plugins_start_timeZresults_to_sendZcustom_metricsZnew_import_metricsZnew_import_valuesr r r Z
schedule_tkeyZ leeway_timer� r[ rb r� ra r` Zplugin_start_timeZt0Zt1Z time_leftZ
start_timerr Zcountermeasures_metadataZfactsZ lucky_dayrM rd Zstaler Zmod_name�modrh r� r" ZpluginZ json_counterZ file_contentZ counter_dataZrequired_fieldsZ
existing_keysrw r� Zcan_access_docker�responseZcommand_resultsrl Zanomalies_to_reportZmetadata_summaryZforce_send_schedules�delayZ result_datarp rq ZMAX_CONTAINERS_SYNCZcontainer_queuer� Zdeleted_container_queueZ dem_resultsZ dem_updatesZdemKey�qZcountermeasurerx Z cm_metadataZ
metadata_filer� r- ra �linesZ
log_outputrz r{ r_ �ss_cmdrk r| Z
parsed_urlZmtr_cmdZ
mtr_outputZ
log_level_keyr� �level�messagerT Zexisting_containersZexisting_container_idsZfound_containers�tr� Zfound_container_idsZnew_containersZ containerZ container_idr r r �main� s�
"
$
&
z
Agent.mainc C s4 | j s
i S yddlm} |� }|j� S i S d S )Nr )�ForticlientHelper)r& Zforticlient_helperr� Zget_handshake_data)r^ r� Zhelperr r r r� � s z Agent.get_fortisase_attributesc C s
dd l }dd l}|j� ��}|j| j|dd� tj� }dj|jd��}t j
j |d�}t j |� |j
t j
j d|�d|�}zny:dj| j�} d j||| �}
t j|
� | jjd
j|�� W n. tk
r� } z| jj| � W Y d d } ~ X nX W d t j
j|�r�t j|� X W d Q R X d S ) Nr T)Z
dirs_exist_okz
agent-logs-{}z %Y%m%d%H%M%S�zipz/tmpz{}/v2/agent_logszHcurl -F file=@{} -H "Accept: application/json" -H "Authorization: {}" {}zUploaded log file {})�shutilr. ZTemporaryDirectoryZcopytreern r rh r� r� r= r> r �mkdirZ make_archiver� ro r@ r� r� r� r rj ) r^ r� r� r. Z
tmpdirnamerh Zzip_file_prefixZ
zip_outputZzip_name�endpointZccr� r r r r� � s0
" zAgent.upload_logsNc C s |r | j jd� | j|d� d S d}| js@||kr<|j|� d S d }|sX| j jd� d S |sttj| j| j || j
d�}d}�y`t dd�}||k�r|| j d kr�dd l
m} |t|j� ��} tj� t | d
� ||< n�yV| j jd �\}
} tj� } t| j| j| jt|
�t| �d �}
|
| k �r|
| }
|
||< W n< tk
�r` } z | j jd
j| j t|��� d S d }~X nX | j jdj|| �� d S || }| �s�tj� |k�r�| j|� |d k�r�tj� }|| ||< | j jdj|| �� W n6 tk
�r } z| j jdj|�� W Y d d }~X nX d S )NzAdmin update request)r� �next_update_checkz checkForUpdate: no server key)r� r5 )Zdaysr )� randrange)ra r4 )�year�month�dayZhourZminutez%Could not calculate next check {}: {}zNext update check at {}zcheckForUpdates problem: {})r@ r� �_onCheckUpdatesr* r� r~ r� r� r� r� r� r r+ r� r� rF r� r rh rE r� r� r� r� r� r� r )r^ r� r� r� r� Zdb_keyr� Z
update_periodr� Z randomSec�hr! ZrnZctr� r r r r� sX
zAgent.checkForUpdatec C s� | j jd� yBdj| j�}|j|dd�}t|�dkrL| jd| jtj |�d� W n4 t
k
r� } z| j j dj|�� W Y d d }~X nX d S ) NzPerforming updates check...zagent_update_info/darwin/{}�GET)�methodr �updates)r] �payloadzUpdate check failure: {}) r@ r� r� r� �callr� r� r9 r r� r� r� )r^ r� r� r� r� r r r r� B s zAgent._onCheckUpdatesc C sX | j jd� g }g }x�t| jd j� �D ]�\}}| jd j|�}|sR| jd |= q(x�t|j� �D ]�\}}| j jd|� |j r�|j|j �r�|j
||t j |j
j� �df� d|_| j jd|� |j o�|j� r`|j
||t j |j
j� �df� d|_| j jd|� q`W q(W | j jd t|�� | j jd
|� | j jd t|�� | j jd |� | jj� || S )
Nz Gathering reportable anomaliesr` r z Threshold %sFTzCleared anomaly: %szLengthy anomaly: %sz$Found %d anomalies that have clearedzCleared anomalies: %rz5Found %d anomalies that exceed the threshold durationzLengthy anomalies: %r)r@ r� r� r� r� r� rA �reported_as_clearedZ has_clearedZnumber_of_checksr� rT r� Ztime_last_detectedr� Z reported_as_exceeded_durationZexceeds_durationr� r� )r^ Zcleared_anomaliesZlengthy_anomalies� schedule_idr` r � threshold_id�anomalyr r r �get_reportable_anomaliesU s@
z Agent.get_reportable_anomaliesc C s� | j jd� xvt| jd j� �D ]`\}}xBt|j� �D ]2\}}|jr6|j|�}| j jd� | j jd|� q6W |s | jd j|� q W | j jd| jd � | jj� d S )Nz'Checking for reported cleared anomaliesr` z Removed reported cleared anomalyz Anomaly: %szRemaining anomalies: %s) r@ r� r� r� r� r� r� rA r� )r^ r� r` r� r� r r r �!remove_reported_cleared_anomalies� s
z'Agent.remove_reported_cleared_anomaliesc C s4 |g ks|d kr | j jd� d S | jd }i | jd< x�|D ]�}|d }| j jd|� |j|d �}y`|r~|j|� ||j= d}n|s�t|�}d}|| jd |j< | j jd||j� | j jd|� W q: tk
r� t j
� d }t |�}| j j d
j
||�� Y q:X q:W | jj� | j jd t|�� | j jd t|�� d S )
Nz,No schedule changes received from aggregatorr r� z$Received schedule %s from aggregatorZEditedZCreatedz%s schedule %s locallyzSchedule data: %rr5 z Invalid schedule {} data: {}z Created/updated %d scheduleszDeleted %d schedules)r@ r� r� r� r� r� r rA r� r� r� r r� r� r� r� ) r^ Z
new_schedulesZexisting_schedulesZnew_schedule_dataZnew_schedule_idr r r� r� r r r r� � s6
zAgent.update_schedulesc C s� d}|d| j 7 }|dtjddd� 7 }dtjkrJ|dtjd dd� 7 }n|dtjd
dd� 7 }|d tjd dd� 7 }dtjkr�|d
tjd| j dd� 7 }|dtjddd� 7 }|dtj|jd| j d� 7 }|S )zVFunction to build a string of diagnostics data to send
back to the aggregator.zAGENT DIAGNOSTICS
zAgent version: %s
zAgent server hostname: %sr� T)ry r� z Agent OS: %sz sw_vers | grep ProductVersionz%cat /etc/*-release | grep PRETTY_NAMEzuname output: %szuname -azPackage information: %s
z apt-cache show %s-agent || truez
ip output:
%sz ip addr showz!Local agent pickle file data:
%s
r1 )rk rN )
r� r# r r� r� rR r r� �data� defaultprint)r^ r� r� rR �stringr r r r� � s
zAgent.build_diagnosticsc C s t |t�r|j� S d S d S )N)�
isinstancer �__repr__)r^ r" r r r r� � s
zAgent.defaultprintc C s� t jj| j�s t jdj| j�� yt| j�}W n
d S i dd dd d | jdd t � i d d t ddd�t ddd�t ddd�t ddd�d �}x(t
|j � �D ]\}}||kr�|||< q�W |S )
Nz mkdir -p {}Fr i� �2 )r{ rz �d rD )r` rB rj re rw rY r� rX rZ rc r ro rn Z
check_resultsZserver_resource_levelsru rv ) r= r> r rp ro r� r r5 rI r r� r� )r^ r� Zdefaultsr� rN r r r r@ � s6
z
Agent.open_dbc C s y|j dd�dkS dS d S )N�topoZ auto_scanr� F)r� )r^ r� r r r �should_run_auto_topo_scans� s z Agent.should_run_auto_topo_scansc
C s yt |jdd��S dS d S )Nr� Zscans_per_syncr )rF r� )r^ r� r r r �get_num_topo_scans� s zAgent.get_num_topo_scansc
C s yt |jdd��S dS d S )Nr� �
scan_sleepr5 )rF r� )r^ r� r r r �get_topo_scan_sleep s zAgent.get_topo_scan_sleepc C sh d}d}t j � }| jjd� yttj|dd��}W n tj� }Y nX t j � | }| jjd| � |S )Nzss -t -u -r 2>&1r� zStarting topo scanT)ry z.Topo scan complete. Elapsed time: %.2f seconds)rT r@ r� r r# r rH rI )r^ r� �resultr� �elapsedr r r �
run_topo_scan
s zAgent.run_topo_scanc C sz | j |�sd S | j|�}| j|�}d| jkr6g | jd< x>t|�D ]2}tj� }| j� }| jd j||f� tj|� q@W d S )Nrl ) r� r� r� r� r� rT r� r� r\ )r^ r� �nr� r� r� �scanr r r r� s
zAgent.run_auto_topo_scansc C s4 | j s
d S | jd| jd�}y
tj|�S d S d S )Nz wifi-info)r] )r7 r� r8 r r )r^ r� r r r r� ) s
zAgent.get_dem_wifi_infoc C s� i }| j s|S | jd| jd�}|d kr*|S tj|�}x�|j� D ]x}y<|| }|j� rb|| ||< n|j|| � |j� ||< W q> t k
r� } z| j
j dj |�� w>W Y d d }~X q>X q>W |S )NZcollect)r] z_getDemResults: {})
r7 r� r8 r r r ZisEmptyr� r� r� r@ r� r� )r^ r� Zrvr� Z
latestResultsr� r� r� r r r r� 6 s$
zAgent._getDemResultsc C sD d}d|kr2|d r2|d | j kr@| j |d< d}n| j |d< d}|S )NFZlast_ran_versionT)r� )r^ r� Z
has_updater r r r� M s
z Agent._agent_version_updatedc C s� yN|j ddd�}|jdg �|jdg �|jdg �d�}| jd| jtj|�d � W n6 tk
r� } ztjd
j t
|��� W Y d d }~X nX d S ) Nr r� )r� rs rt rv )rs rt rv Z
initSchedules)r] r� z/schedules error: {}) r� r� r� r8 r r� r� r; r� r� r )r^ Zclientr� r ZaggExr r r r� Y s
&zAgent._init_dem_schedulesc C s" | j r | jd| jtj|�d�}d S )Nzupdate-schedules)r] r� )r7 r� r8 r r� )r^ Z newSchedulesrm r r r r� g s z Agent._updateDEMServiceSchedulesc C s� |s t d��dd l}|r$dj||�}|d7 }y�|j|j|j��|}|jd� |jtjj | j
|�� |j d�}|j |� t
� }x$|jd�}dt|�kr�P ||7 }q|W |jd�} d| kr�d S | S Q R X W n4 t k
� r� }
z| jjd j|
�� d S d }
~
X nX d S )
NzServer port not configuredr z{}:{}r� g $@zutf-8i ZnackzSend/recv failure: {})r� �socketr� ZAF_UNIXZ SOCK_STREAMZ
settimeoutZconnectr= r> r r: r� Zsendall�bytesZrecvr� r� r@ r� ) r^ ri r] r� r� ZsockZtoSendZ
receivedBytesry Zreceivedr� r r r r� l s2
zAgent._sendReceive)r1 r2 r3 )F)r� )NNF)NN)2r r r r� rI r r] ri rS rY r� r� r� r� r� r� r� r� r r r r# r' rA r� r? r= rX r� r� r� r� r� r� r� r� r� r� r@ r� r� r� r� r� r� r� r� r� r� r� r r r r r. [ sl
D
&P
t=
% "'
/-#+
r. )PZ
agent_utilr r r r Z inspectorr r r Zagent_exceptionsr r
Zprocess_managerr Zos.pathr r
r r r Zpickle_databaser Zplugin_managerr r Zprogress_printerr rc r r r r� r r� �ior� r r� r� r� r� r� r� r; Zlogging.handlersrV r= r� r� r# r. rT rH r� r� Z blacklisterr r� r r{ rz r � ImportErrorZ
simplejsonZhashlibZsha1r� Zsha�newr� r- r r� Z urllib.parse�parseZurllib.requestZrequestr� �objectr. r r r r �<module> s�