§
    -P„jx  ã                   óÌ  — d Z ddlmZ ddlmZ ddlmZ ddlZddlZddlZddlm	Z	 ddl
mZ ddlmZ dd	lmZ dd
lmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddlmZ ddl m!Z" ddl m#Z$ ddl m%Z% ddl m&Z& ddl m'Z' g d¢Z(d„ Z)d„ Z*d„ Z+dZ,e,dddde"j-        j.        ddddf
d „Z/d!„ Z0d"„ Z1dZ2e1e2d#dd$ddde2de2de2e2d%e2ddd%dddddfd&„Z3dS )'aÜ   Contains TF-Slim code for training models.

This script contains various functions for training models. These include
manipulating gradients, creating a `train_op` (an operation that computes the
loss and applies the gradients) and a training loop function. The training loop
allows the user to pass in the `train_op` and runs the optimization according
to user-specified arguments. Note that the training loop uses the
tf.compat.v1.train.Supervisor and its managed_session in its implementation to
ensure the
ability of worker processes to recover from failures.

************************************
* A simple working training script *
************************************

  # Load data and create the model:
  images, labels = LoadData(...)
  predictions = MyModel(images)

  # Define the loss:
  slim.losses.log_loss(predictions, labels)
  total_loss = slim.losses.get_total_loss()

  # Define the optimizer:
  optimizer = tf.compat.v1.train.MomentumOptimizer(FLAGS.learning_rate,
  FLAGS.momentum)

  # Create the train_op
  train_op = slim.learning.create_train_op(total_loss, optimizer)

  # Run training.
  slim.learning.train(train_op, my_log_dir)

*************************
* Creating the train_op *
*************************

In order to train, TF-Slim's train loop needs a train_op: an `Operation` that
(a) computes the loss, (b) applies the gradients to update the weights and
(c) returns the value of the loss. slim.learning.create_train_op creates
such an `Operation`. This function also provides the ability to manipulate
the gradients using a few arguments:

  # Create the train_op and clip the gradient norms:
  train_op = slim.learning.create_train_op(
      total_loss,
      optimizer,
      clip_gradient_norm=4)

  # Create the train_op and scale the gradients by providing a map from variable
  # name (or variable) to a scaling coefficient:
  gradient_multipliers = {
    'conv0/weights': 1.2,
    'fc8/weights': 3.4,
  }
  train_op = slim.learning.create_train_op(
      total_loss,
      optimizer,
      gradient_multipliers=gradient_multipliers)

****************************************************************
* Performing additional (non-gradient) updates during training *
****************************************************************

Many networks utilize modules, like BatchNorm, that require performing a series
of non-gradient updates during training. slim.learning.create_train_op allows
a user to pass in a list of update_ops to call along with the gradient updates.

  train_op = slim.learning.create_train_op(total_loss, optimizer, update_ops)

By default, slim.learning.create_train_op includes all update ops that are
part of the `tf.GraphKeys.UPDATE_OPS` collection. Additionally, TF-Slim's
slim.batch_norm function adds the moving mean and moving variance updates to
this collection. Consequently, users who want to use slim.batch_norm will not
need to take any additional steps in order to have the moving mean and moving
variance updates be computed.

However, users with additional, specialized updates can either override the
default update ops or simply add additional update ops to the
`tf.GraphKeys.UPDATE_OPS` collection:

  # Force TF-Slim NOT to use ANY update_ops:
  train_op = slim.learning.create_train_op(
     total_loss,
     optimizer,
     update_ops=[])

  # Use an alternative set of update ops:
  train_op = slim.learning.create_train_op(
     total_loss,
     optimizer,
     update_ops=my_other_update_ops)

  # Use an alternative set of update ops in addition to the default updates:
  tf.compat.v1.add_to_collection(tf.GraphKeys.UPDATE_OPS, my_update0)
  tf.compat.v1.add_to_collection(tf.GraphKeys.UPDATE_OPS, my_update1)

  train_op = slim.learning.create_train_op(
     total_loss,
     optimizer)

  # Which is the same as:
  train_op = slim.learning.create_train_op(
     total_loss,
     optimizer,
     update_ops=tf.compat.v1.get_collection(tf.GraphKeys.UPDATE_OPS))

******************************************
* Initializing a model from a checkpoint *
******************************************

It is common to want to 'warm-start' a model from a pre-trained checkpoint.
TF-Slim provides a convenient mechanism for doing so:

  ...

  # Create the train_op
  train_op = slim.learning.create_train_op(total_loss, optimizer)

  # Create the initial assignment op
  checkpoint_path = '/path/to/old_model_checkpoint'
  variables_to_restore = slim.get_model_variables()
  init_assign_op, init_feed_dict = slim.assign_from_checkpoint(
      checkpoint_path, variables_to_restore)

  # Create an initial assignment function.
  def InitAssignFn(sess):
      sess.run(init_assign_op, init_feed_dict)

  # Run training.
  slim.learning.train(train_op, my_log_dir, init_fn=InitAssignFn)

***************************************************************************
* Initializing a model from a checkpoint whose variable names don't match *
***************************************************************************

At times, a user may want to initialize a new model with values from a
checkpoint whose variable names do not match those of the current model. In this
case, one needs to create a mapping from the checkpoint variable names to the
current model variables. This requires only a small modification of the code
above:
  ...
  # Creates a model with two variables, var0 and var1
  predictions = MyModel(images)
  ...

  # Create the train_op
  train_op = slim.learning.create_train_op(total_loss, optimizer)

  checkpoint_path = '/path/to/old_model_checkpoint'

  # Create the mapping:
  variables_to_restore = {
      'name_var_0_in_checkpoint': slim.get_unique_variable('var0'),
      'name_var_1_in_checkpoint': slim.get_unique_variable('var1')
  }
  init_assign_op, init_feed_dict = slim.assign_from_checkpoint(
      checkpoint_path, variables_to_restore)

  # Create an initial assignment function.
  def InitAssignFn(sess):
      sess.run(init_assign_op, init_feed_dict)

  # Run training.
  slim.learning.train(train_op, my_log_dir, init_fn=InitAssignFn)


*************************************************
* Fine-Tuning Part of a model from a checkpoint *
*************************************************

Rather than initializing all of the weights of a given model, we sometimes
only want to restore some of the weights from a checkpoint. To do this, one
need only filter those variables to initialize as follows:

  ...

  # Create the train_op
  train_op = slim.learning.create_train_op(total_loss, optimizer)

  checkpoint_path = '/path/to/old_model_checkpoint'

  # Specify the variables to restore via a list of inclusion or exclusion
  # patterns:
  variables_to_restore = slim.get_variables_to_restore(
      include=["conv"], exclude=["fc8", "fc9])
  # or
  variables_to_restore = slim.get_variables_to_restore(exclude=["conv"])

  init_assign_op, init_feed_dict = slim.assign_from_checkpoint(
      checkpoint_path, variables_to_restore)

  # Create an initial assignment function.
  def InitAssignFn(sess):
      sess.run(init_assign_op, init_feed_dict)

  # Run training.
  slim.learning.train(train_op, my_log_dir, init_fn=InitAssignFn)

******************************************************
* Initializing model variables from values in memory *
******************************************************

One may want to initialize the weights of a model from values from an arbitrary
source (a text document, matlab file, etc). While this is technically feasible
using plain TensorFlow, it also results in the values of your weights being
stored in the graph. For large models, this becomes prohibitively large. TF-Slim
allows you to perform this initial assignment without having to store the values
of the initial model in the graph itself by using placeholders and a feed
dictionary:

  ...

  # Create the train_op
  train_op = slim.learning.create_train_op(total_loss, optimizer)

  # Create the mapping from variable names to values:
  var0_initial_value = ReadFromDisk(...)
  var1_initial_value = ReadFromDisk(...)

  var_names_to_values = {
    'var0': var0_initial_value,
    'var1': var1_initial_value,
  }
  init_assign_op, init_feed_dict = slim.assign_from_values(var_names_to_values)

  # Create an initial assignment function.
  def InitAssignFn(sess):
      sess.run(init_assign_op, init_feed_dict)

  # Run training.
  slim.learning.train(train_op, my_log_dir, init_fn=InitAssignFn)
é    )Úabsolute_import)Údivision)Úprint_functionN)Útraining)Ú
config_pb2)Útimeline)Úconstant_op)Úerrors)Úops)Úfile_io)Ú	array_ops)Úclip_ops)Úcontrol_flow_ops)Ú
lookup_ops)Úmath_ops)Ú	variables)Ú
tf_logging)Úsummary)Ú	optimizer)Úsaver)Ú
supervisor)Úsync_replicas_optimizer)Útraining_util)Úadd_gradients_summariesÚclip_gradient_normsÚmultiply_gradientsÚcreate_train_opÚ
train_stepÚtrainc                 ó   — g }| D ]ˆ\  }}|�jt          |t          j        ¦  «        r;t          j        |j        |¦  «        }t          j        ||j        |j        ¦  «        }nt          j        ||¦  «        }|                     ||f¦  «         Œ‰|S )zâClips the gradients by the given value.

  Args:
    gradients_to_variables: A list of gradient to variable pairs (tuples).
    max_norm: the maximum norm value.

  Returns:
    A list of clipped gradient to variable pairs.
  )	Ú
isinstancer   ÚIndexedSlicesr   Úclip_by_normÚvaluesÚindicesÚdense_shapeÚappend)Úgradients_to_variablesÚmax_normÚclipped_grads_and_varsÚgradÚvarÚtmps         úU/root/disk_3tb/conda_envs/Huy-visual/lib/python3.11/site-packages/tf_slim/learning.pyr   r     sš   € ð ÐØ)ð /ð /�i€dˆCØÐÝ	�D�#Ô+Ñ	,Ô	,ð 5ÝÔ# D¤K°Ñ:Ô:ˆÝÔ   d¤l°DÔ4DÑEÔEˆˆåÔ$ T¨8Ñ4Ô4ˆØ×!Ò! 4¨ +Ñ.Ô.Ð.Ð.Ø	Ðó    c                 ór  — t          | t          ¦  «        st          d¦  «        ‚|st          d¦  «        ‚t          |t          ¦  «        st          d¦  «        ‚g }| D ]Ø\  }}||v s|j        j        |v rª||v r|n|j        j        }|€t          d¦  «        ‚||         }t          |t          j        ¦  «        st          j	        ||j
        ¬¦  «        }t          |t          j        ¦  «        r+|j        |z  }t          j        ||j        |j        ¦  «        }n||z  }|                     ||f¦  «         ŒÙ|S )aè  Multiply specified gradients.

  Args:
    grads_and_vars: A list of gradient to variable pairs (tuples).
    gradient_multipliers: A map from either `Variables` or `Variable` op names
      to the coefficient by which the associated gradient should be scaled.

  Returns:
    The updated list of gradient to variable pairs.

  Raises:
    ValueError: If `grads_and_vars` is not a list or if `gradient_multipliers`
    is empty or None or if `gradient_multipliers` is not a dictionary.
  z `grads_and_vars` must be a list.z `gradient_multipliers` is empty.z&`gradient_multipliers` must be a dict.Nz&Requested multiple of `None` gradient.)Údtype)r!   ÚlistÚ
ValueErrorÚdictÚopÚnamer   ÚTensorr	   Úconstantr1   r"   r$   r%   r&   r'   )Úgrads_and_varsÚgradient_multipliersÚmultiplied_grads_and_varsr+   r,   ÚkeyÚ
multiplierr-   s           r.   r   r   5  s[  € õ 
�N¥DÑ	)Ô	)ð 9Ý
Ð7Ñ
8Ô
8Ð8Ø	ð 9Ý
Ð7Ñ
8Ô
8Ð8Ý	Ð(­$Ñ	/Ô	/ð ?Ý
Ð=Ñ
>Ô
>Ð>à ÐØ!ð 2ð 2�i€dˆCØ
Ð"Ð"Ð" c¤f¤kÐ5IÐ&IÐ&IØÐ.Ð.Ð.ˆCˆC°C´F´K€cØ	ˆÝÐAÑBÔBÐBà'¨Ô,€jÝ˜
¥C¤JÑ/Ô/ð HÝ Ô)¨*¸D¼JÐGÑGÔGˆ
å	�D�#Ô+Ñ	,Ô	,ð ØŒk˜JÑ&ˆÝÔ   d¤l°DÔ4DÑEÔEˆˆà�
ÑˆØ×$Ò$ d¨C [Ñ1Ô1Ð1Ð1Ø	"Ð"r/   c           	      ó   — g }| D ]È\  }}|�¢t          |t          j        ¦  «        r|j        }n|}|                     t          j        |j        j        dz   |¦  «        ¦  «         |                     t          j	        |j        j        dz   t          j        |g¦  «        ¦  «        ¦  «         Œ©t          j        d|j        j        ¦  «         ŒÉ|S )z™Add summaries to gradients.

  Args:
    grads_and_vars: A list of gradient to variable pairs (tuples).

  Returns:
    The list of created summaries.
  Nz	/gradientz/gradient_normzVar %s has no gradient)r!   r   r"   r$   r'   r   Ú	histogramr5   r6   Úscalarr   Úglobal_normÚloggingÚinfo)r9   Ú	summariesr+   r,   Úgrad_valuess        r.   r   r   _  sã   € ð €)Ø!ð :ð :�i€dˆCØÐÝ	�D�#Ô+Ñ	,Ô	,ð Ø”kˆˆàˆØ×ÒÝ
Ô
˜CœFœK¨+Ñ5°{Ñ
CÔ
CñEô Eð Eà×ÒÝ
Œ.˜œœÐ'7Ñ7Ý!Ô-¨{¨mÑ<Ô<ñ>ô >ñ?ô ?ð ?ð ?õ „lÐ+¨S¬V¬[Ñ9Ô9Ð9Ð9à	Ðr/   FTc                 óP   ‡‡
— ˆˆ
fd„}t          j        | |||||||||	|¬¦  «        S )a|  Creates an `Operation` that evaluates the gradients and returns the loss.

  Args:
    total_loss: A `Tensor` representing the total loss.
    optimizer: A tf.Optimizer to use for computing the gradients.
    global_step: A `Tensor` representing the global step variable. If left as
      `_USE_GLOBAL_STEP`, then tf.train.global_step() is used.
    update_ops: An optional list of updates to execute. If `update_ops` is
      `None`, then the update ops are set to the contents of the
      `tf.GraphKeys.UPDATE_OPS` collection. If `update_ops` is not `None`, but
      it doesn't contain all of the update ops in `tf.GraphKeys.UPDATE_OPS`, a
      warning will be displayed.
    variables_to_train: an optional list of variables to train. If None, it will
      default to all tf.compat.v1.trainable_variables().
    clip_gradient_norm: If greater than 0 then the gradients would be clipped by
      it.
    summarize_gradients: Whether or not add summaries for each gradient.
    gate_gradients: How to gate the computation of gradients. See tf.Optimizer.
    aggregation_method: Specifies the method used to combine gradient terms.
      Valid values are defined in the class `AggregationMethod`.
    colocate_gradients_with_ops: Whether or not to try colocating the gradients
      with the ops that generated them.
    gradient_multipliers: A dictionary of either `Variables` or `Variable` op
      names to the coefficient by which the associated gradient should be
      scaled.
    check_numerics: Whether or not we apply check_numerics.

  Returns:
    A `Tensor` that when evaluated, computes the gradients and returns the total
      loss value.
  c                 ó  •— ‰r<t          j        d¦  «        5  t          | ‰¦  «        } d d d ¦  «         n# 1 swxY w Y   ‰dk    r<t          j        d¦  «        5  t          | ‰¦  «        } d d d ¦  «         n# 1 swxY w Y   | S )NÚmultiply_gradsr   Ú
clip_grads)r   Ú
name_scoper   r   )ÚgradsÚclip_gradient_normr:   s    €€r.   Útransform_grads_fnz+create_train_op.<locals>.transform_grads_fn©  s  ø€ Øð @ÝŒ>Ð*Ñ+Ô+ð @ð @Ý" 5Ð*>Ñ?Ô?ˆð@ð @ð @ñ @ô @ð @ð @ð @ð @ð @ð @øøøð @ð @ð @ð @ð ˜AÒÐÝŒ>˜,Ñ'Ô'ð ?ð ?Ý# EÐ+=Ñ>Ô>ˆð?ð ?ð ?ñ ?ô ?ð ?ð ?ð ?ð ?ð ?ð ?øøøð ?ð ?ð ?ð ?à€Ls   ˜5µ9¼9ÁA7Á7A;Á>A;)Ú
total_lossr   Úglobal_stepÚ
update_opsÚvariables_to_trainrM   Úsummarize_gradientsÚgate_gradientsÚaggregation_methodÚcolocate_gradients_with_opsÚcheck_numerics)r   r   )rN   r   rO   rP   rQ   rL   rR   rS   rT   rU   r:   rV   rM   s        `    `  r.   r   r   }  s`   øø€ ðX	ð 	ð 	ð 	ð 	ð 	õ 
Ô	!ØØØØØ+Ø+Ø-Ø#Ø+Ø"=Ø#ð
%ñ 
%ô 
%ð %r/   c                 ód   — 	 t          j        | |¦  «        |k    rdS t          j        d¦  «         Œ0)zœWait till the global step has reached at least 'step'.

  Args:
    sess: A session.
    global_step: A Tensor.
    step: Int.  The global step to reach.
  Tg      ð?N)r   rO   ÚtimeÚsleep)ÚsessrO   Ústeps      r.   Ú_wait_for_stepr\   Â  s6   € ðÝÔ   {Ñ3Ô3°tÒ;Ð;Ø€eÝ„Jˆs�O„O€Oðr/   c                 ój  — t          j         ¦   «         }d}d}d|v red|vrt          d¦  «        ‚|                      |d         ¦  «        r7t          j        t          j        j        ¬¦  «        }t          j        ¦   «         }|                      ||g||¬¦  «        \  }}t          j         ¦   «         |z
  }	|�£t          j        |j	        ¦  «        }
|
 
                    ¦   «         }t          j                             |d         d|z  ¦  «        }t          j        d|¦  «         t!          j        ||¦  «         d	|v r|d	                              |d
|z  ¦  «         d|v r2|                      |d         ¦  «        rt          j        d|||	¦  «         d|v r|                      |d         ¦  «        }nd}||fS )aû  Function that takes a gradient step and specifies whether to stop.

  Args:
    sess: The current session.
    train_op: An `Operation` that evaluates the gradients and returns the total
      loss.
    global_step: A `Tensor` representing the global training step.
    train_step_kwargs: A dictionary of keyword arguments.

  Returns:
    The total loss and a boolean indicating whether or not to stop training.

  Raises:
    ValueError: if 'should_trace' is in `train_step_kwargs` but `logdir` is not.
  NÚshould_traceÚlogdirzHlogdir must be present in train_step_kwargs when should_trace is present)Útrace_level)ÚoptionsÚrun_metadataztf_trace-%d.jsonzWriting trace to %sÚsummary_writerzrun_metadata-%dÚ
should_logz+global step %d: loss = %.4f (%.3f sec/step)Úshould_stopF)rX   r3   Úrunr   Ú
RunOptionsÚ
FULL_TRACEÚRunMetadatar   ÚTimelineÚ
step_statsÚgenerate_chrome_trace_formatÚosÚpathÚjoinrB   rC   r   Úwrite_string_to_fileÚadd_run_metadata)rZ   Útrain_oprO   Útrain_step_kwargsÚ
start_timeÚtrace_run_optionsrb   rN   Únp_global_stepÚtime_elapsedÚtlÚtraceÚtrace_filenamere   s                 r.   r   r   Ð  sù  € õ  Œy‰{Œ{€*àÐØ€,ØÐ(Ð(Ð(ØÐ(Ð(Ð(Ýð 1ñ 2ô 2ð 2à‡x‚xÐ! .Ô1Ñ2Ô2ð .Ý$Ô/Ý Ô+Ô6ð8ñ 8ô 8ÐåÔ+Ñ-Ô-€là#Ÿxšx¨°;Ð(?Ø0AØ5Að  (ñ  Cô  CÑ€*ˆnõ ”‘”˜zÑ)€,àÐÝ	Ô	˜<Ô2Ñ	3Ô	3€BØ×+Ò+Ñ-Ô-€EÝ”W—\’\Ð"3°HÔ"=Ø"4°~Ñ"EñGô G€Nå„LÐ&¨Ñ7Ô7Ð7ÝÔ  °Ñ7Ô7Ð7ØÐ,Ð,Ð,ØÐ(Ô)×:Ò:Ø
Ð)¨NÑ:ñ<ô <ð <ð Ð&Ð&Ð&Ø‡x‚xÐ! ,Ô/Ñ0Ô0ð =Ý„lÐ@Ø! :¨|ñ=ô =ð =ð Ð'Ð'Ð'Ø—(’(Ð,¨]Ô;Ñ<Ô<€K€Kà€Kà	�[Ð	 Ð r/   é   Ú iX  c                 ó  — | €t          d¦  «        ‚|€<|t          k    rt          d¦  «        ‚|�t          d¦  «        ‚|�t          d¦  «        ‚t          |t          j        ¦  «        r|g}|�|dk    rt          d¦  «        ‚|	�|	dk    rt          d¦  «        ‚|pt          j        ¦   «         }|                     ¦   «         5  |€t          j	        ¦   «         }|pt          j        ¦   «         }|�.|D ]+}t          |t          j        ¦  «        st          d	¦  «        ‚Œ,t          j        d
¦  «        5  |
t          k    rt          j        ¦   «         }
|t          k    rt          j        ¦   «         }|t          k    r7t!          j        t          j        ¦   «         t'          j        ¦   «         ¦  «        }|�št          |t*          ¦  «        r…t          j        |�|gng ¦  «        5  |rt!          j        d„ |D ¦   «         Ž }nt!          j        d„ |D ¦   «         Ž }ddd¦  «         n# 1 swxY w Y   t/          j        d„ |D ¦   «         d¬¦  «        }nd}ddd¦  «         n# 1 swxY w Y   |t          k    rt3          j        ¦   «         }|t          k    rt6          j        j        }|r|�d„ |D ¦   «         }d„ |D ¦   «         }|t          k    rÄt          j        d¦  «        5  i }|	rt=          j        ||	¦  «        }ntA          j!        d¦  «        }||d<   |dk    r+t=          j"        t=          j#        ||¦  «        d¦  «        |d<   |r2|�0t=          j"        t=          j#        ||¦  «        d¦  «        |d<   ||d<   ddd¦  «         n# 1 swxY w Y   ddd¦  «         n# 1 swxY w Y   t7          j        ||||
|||||||||||¬¦  «        }|�
|j$        |d<   d} d}!|!�r.	 d}!| %                    |d|¬¦  «        5 }"tM          j'        d¦  «         |� tM          j'        d|¦  «          ||"¦  «        }"|r|r| (                    |"¦  «         n1|dk    r+tS          |"|tU          ||	ptV          j,        ¦  «        ¦  «         | -                    |"¦  «        }#tM          j'        d¦  «         |r-|�+| -                    |"|¦  «         |" .                    |¦  «         	 | /                    ¦   «         sP ||"| ||¦  «        \  } }$|$r)tM          j'        d¦  «         | 0                    ¦   «          n| /                    ¦   «         ¯Pn1# tb          j2        $ r}%tM          j'        d|%¦  «         Y d}%~%nd}%~%ww xY w|rZ|j3        rStM          j'        d ¦  «         |j4         5                    |"|j6        |j7        ¬!¦  «         | 8                    |#d|¬"¦  «         ddd¦  «         n# 1 swxY w Y   n+# tb          j9        $ r tM          j'        d#¦  «         d}!Y nw xY w|!�°.| S )$a`  Runs a training loop using a TensorFlow supervisor.

  When the sync_optimizer is supplied, gradient updates are applied
  synchronously. Otherwise, gradient updates are applied asynchronous.

  Args:
    train_op: A `Tensor` that, when executed, will apply the gradients and
      return the loss value.
    logdir: The directory where training logs are written to. If None, model
      checkpoints and summaries will not be written.
    train_step_fn: The function to call in order to execute a single gradient
      step. The function must have take exactly four arguments: the current
        session, the `train_op` `Tensor`, a global step `Tensor` and a
        dictionary.
    train_step_kwargs: A dictionary which is passed to the `train_step_fn`. By
      default, two `Boolean`, scalar ops called "should_stop" and "should_log"
      are provided.
    log_every_n_steps: The frequency, in terms of global steps, that the loss
      and global step are logged.
    graph: The graph to pass to the supervisor. If no graph is supplied the
      default graph is used.
    master: The address of the tensorflow master.
    is_chief: Specifies whether or not the training is being run by the primary
      replica during replica training.
    global_step: The `Tensor` representing the global step. If left as `None`,
      then training_util.get_or_create_global_step(), that is,
      tf.train.global_step() is used.
    number_of_steps: The max number of gradient steps to take during training,
      as measured by 'global_step': training will stop if global_step is greater
        than 'number_of_steps'. If the value is left as None, training proceeds
        indefinitely.
    init_op: The initialization operation. If left to its default value, then
      the session is initialized by calling
      `tf.compat.v1.global_variables_initializer()`.
    init_feed_dict: A feed dictionary to use when executing the `init_op`.
    local_init_op: The local initialization operation. If left to its default
      value, then the session is initialized by calling
      `tf.compat.v1.local_variables_initializer()` and
      `tf.compat.v1.tables_initializer()`.
    init_fn: An optional callable to be executed after `init_op` is called. The
      callable must accept one argument, the session being initialized.
    ready_op: Operation to check if the model is ready to use. If left to its
      default value, then the session checks for readiness by calling
      `tf.compat.v1.report_uninitialized_variables()`.
    summary_op: The summary operation.
    save_summaries_secs: How often, in seconds, to save summaries.
    summary_writer: `SummaryWriter` to use.  Can be `None` to indicate that no
      summaries should be written. If unset, we create a SummaryWriter.
    startup_delay_steps: The number of steps to wait for before beginning. Note
      that this must be 0 if a sync_optimizer is supplied.
    saver: Saver to save checkpoints. If None, a default one will be created and
      used.
    save_interval_secs: How often, in seconds, to save the model to `logdir`.
    sync_optimizer: an instance of tf.compat.v1.train.SyncReplicasOptimizer, or
      a list of them. If the argument is supplied, gradient updates will be
      synchronous. If left as `None`, gradient updates will be asynchronous.
    session_config: An instance of `tf.compat.v1.ConfigProto` that will be used
      to configure the `Session`. If left as `None`, the default will be used.
    session_wrapper: A function that takes a `tf.compat.v1.Session` object as
      the only argument and returns a wrapped session object that has the same
      methods that the original object has, or `None`. Iff not `None`, the
      wrapped object will be used for training.
    trace_every_n_steps: produce and save a `Timeline` in Chrome trace format
      and add it to the summaries every `trace_every_n_steps`. If None, no trace
      information will be produced or saved.
    ignore_live_threads: If `True` ignores threads that remain running after a
      grace period when stopping the supervisor, instead of raising a
      RuntimeError.

  Returns:
    the value of the loss function after training.

  Raises:
    ValueError: if `train_op` is empty or if `startup_delay_steps` is
      non-zero when `sync_optimizer` is supplied, if `number_of_steps` is
      negative, or if `trace_every_n_steps` is not `None` and no `logdir` is
      provided.
  Nztrain_op cannot be None.z-Cannot provide summary_op because logdir=Nonez(Cannot provide saver because logdir=Nonez6Cannot provide trace_every_n_steps because logdir=Noner   zAstartup_delay_steps must be zero when sync_optimizer is supplied.z;`number_of_steps` must be either None or a positive number.z:`sync_optimizer` must be a tf.train.SyncReplicasOptimizer.Úinit_opsc                 ó   — g | ]	}|j         ‘Œ
S © )Úchief_init_op©Ú.0Úopts     r.   ú
<listcomp>ztrain.<locals>.<listcomp>¨  s   € Ð>Ð>Ð>¨�#Ô#Ð>Ð>Ð>r/   c                 ó   — g | ]	}|j         ‘Œ
S r€   )Úlocal_step_init_opr‚   s     r.   r…   ztrain.<locals>.<listcomp>«  s   € ÐCÐCÐC¨S�#Ô(ÐCÐCÐCr/   c                 ó   — g | ]	}|j         ‘Œ
S r€   )Úready_for_local_init_opr‚   s     r.   r…   ztrain.<locals>.<listcomp>­  s   € ÐCÐCÐC¨SˆSÔ(ÐCÐCÐCr/   )Úaxisc                 ó6   — g | ]}|                      ¦   «         ‘ŒS r€   )Úget_init_tokens_opr‚   s     r.   r…   ztrain.<locals>.<listcomp>¹  s$   € ÐKÐKÐK°S˜×.Ò.Ñ0Ô0ÐKÐKÐKr/   c                 ó6   — g | ]}|                      ¦   «         ‘ŒS r€   )Úget_chief_queue_runnerr‚   s     r.   r…   ztrain.<locals>.<listcomp>º  s1   € ð ð ð Ø+.ˆ#×
$Ò
$Ñ
&Ô
&ðð ð r/   r   Fre   rd   r^   r_   )ÚgraphÚis_chiefr_   Úinit_opÚinit_feed_dictÚlocal_init_opr‰   Úready_opÚ
summary_oprc   rO   r   Úsave_summaries_secsÚsave_model_secsÚinit_fnrc   T)Ústart_standard_servicesÚconfigzStarting Session.z*Wrapping session with wrapper function: %szStarting Queues.zStopping Training.z-Caught OutOfRangeError. Stopping Training. %sz(Finished training! Saving model to disk.)rO   )Úclose_summary_writerÚignore_live_threadszRetrying training!):r3   Ú_USE_DEFAULTr!   r   ÚSyncReplicasOptimizerr   Úget_default_graphÚ
as_defaultr   Úget_or_create_global_stepÚtf_saverÚSaverrJ   r   Úglobal_variables_initializerÚreport_uninitialized_variablesr   ÚgroupÚlocal_variables_initializerr   Útables_initializerr2   Úcontrol_dependenciesr   Úconcatr   Ú	merge_allr   Ú
SupervisorÚUSE_DEFAULTr   Úgreater_equalr	   r8   ÚequalÚmodrc   Úmanaged_sessionrB   rC   r™   r\   ÚminÚsysÚmaxsizeÚstart_queue_runnersrf   re   Úrequest_stopr
   ÚOutOfRangeErrorr�   r   ÚsaveÚ	save_pathrO   ÚstopÚAbortedError)&rr   r_   Útrain_step_fnrs   Úlog_every_n_stepsr�   Úmasterr�   rO   Únumber_of_stepsr‘   r’   r“   r˜   r”   r•   r–   rc   Ústartup_delay_stepsr   Úsave_interval_secsÚsync_optimizerÚsession_configÚsession_wrapperÚtrace_every_n_stepsrœ   r„   r‰   Úinit_tokens_opÚchief_queue_runnerÚshould_stop_opÚsvrN   Úshould_retryrZ   Úthreadsre   Úes&                                         r.   r   r     sÕ  € ðP ÐÝ
Ð/Ñ
0Ô
0Ð0à€^Ø•\Ò!Ð!ÝÐFÑGÔGÐGØÐÝÐAÑBÔBÐBØÐ&Ýð %ñ &ô &ð &õ �Õ 7Ô MÑNÔNð &Ø$Ð%€NØÐÐ$7¸!Ò$;Ð$;Ý
ØKñMô Mð Mð Ð  _¸Ò%9Ð%9Ý
ØEñGô Gð Gð Ð
*•3Ô(Ñ*Ô*€%Ø×ÒÑÔð A/ð A/ØÐÝ!Ô;Ñ=Ô=€kØÐ%•X”^Ñ%Ô%€EàÐ!Øð Lð Lˆ#Ý˜#Õ6ÔLÑMÔMð 	LÝØJñLô Lð Lð	Lõ 
Œ˜
Ñ	#Ô	#ð 'ð 'Ø	•LÒ	 Ð	 ÝÔ8Ñ:Ô:ˆà	•\Ò	!Ð	!ÝÔ;Ñ=Ô=ˆà	�,Ò	&Ð	&Ý(Ô.ÝÔ1Ñ3Ô3ÝÔ)Ñ+Ô+ñ-ô -ˆð 
Ð	#­
°>Å4Ñ(HÔ(HÐ	#ÝÔ%Ø,Ð8ˆ]ˆOˆO¸bñBô Bð 	Eð 	Eàð EÝ,Ô2Ø>Ð>¨~Ð>Ñ>Ô>ð@ˆMˆMõ -Ô2ØCÐC°NÐCÑCÔCðEˆMð	Eð 	Eð 	Eñ 	Eô 	Eð 	Eð 	Eð 	Eð 	Eð 	Eð 	Eøøøð 	Eð 	Eð 	Eð 	Eõ #,Ô"2ØCÐC°NÐCÑCÔCÈ!ð#Mñ #Mô #MÐÐð #'Ðð1'ð 'ð 'ñ 'ô 'ð 'ð 'ð 'ð 'ð 'ð 'øøøð 'ð 'ð 'ð 'ð4 •\Ò!Ð!ÝÔ$Ñ&Ô&€jà�Ò%Ð%Ý!Ô,Ô8€nàð �NÐ.àKÐK¸NÐKÑKÔK€nðð Ø2@ðñ ô Ðð �LÒ(Ð(ÝŒ>˜,Ñ'Ô'ð /ð /ØÐàð 	7Ý#Ô1°+¸ÑOÔOˆ.ˆ.å&Ô/°Ñ6Ô6ˆ.Ø+9Ð˜-Ñ(Ø˜qÒ Ð Ý,4¬NÝŒl˜;Ð(9Ñ:Ô:¸Añ-?ô -?Ð
˜LÑ
)àð 	/Ð+Ð7Ý.6¬nÝŒl˜;Ð(;Ñ<Ô<¸añ/Aô /AÐ
˜NÑ
+à(.Ð
˜HÑ
%ð/ð /ð /ñ /ô /ð /ð /ð /ð /ð /ð /øøøð /ð /ð /ð /ðgA/ð A/ð A/ñ A/ô A/ð A/ð A/ð A/ð A/ð A/ð A/øøøð A/ð A/ð A/ð A/õF ÔØØØØØ#Ø!Ø5ØØØ#ØØØ-Ø(Øðñ ô €"ð" ÐØ*,Ô*;ÐÐ&Ñ'à€*Ø€,Øñ /ð.Ø€lØ×ÒØ
¨%¸ð ñ Hô Hð &7ØKOÝŒÐ(Ñ)Ô)Ð)ØÐ&Ý
Œ,ÐCØ&ñ(ô (ð (à � Ñ&Ô&ˆ$Øð 	HØð -Ø×&Ò& tÑ,Ô,Ð,øØ  1Ò$Ð$å
Ø�KÝÐ% Ð'E½#¼+ÑFÔFñHô Hð Hð ×(Ò(¨Ñ.Ô.ˆÝŒÐ'Ñ(Ô(Ð(Øð 	#˜Ð2Ø
×
 Ò
  Ð'9Ñ
:Ô
:Ð
:Ø
�(Š(�>Ñ
"Ô
"Ð
"ð	KØ—N’NÑ$Ô$ð Ø&3 m°D¸(ÀKØ4Eñ'Gô 'GÑ#ˆJ˜àð ÝŒlÐ/Ñ0Ô0Ð0Ø�oŠoÑÔÐØð —N’NÑ$Ô$ð øøõ Ô%ð 	Kð 	Kð 	Kõ Œ,ÐFÈÑ
JÔ
JÐ
JÐ
JÐ
JÐ
JÐ
JÐ
Jøøøøð	Køøøð ð 	7�b”kð 	7Ý
Œ,ÐAÑ
BÔ
BÐ
BØ
Œ(�-Š-˜˜bœl¸¼ˆ-Ñ
GÔ
GÐ
GØ
�'Š'ØØ#'Ø"5ð ñ 7ô 7ð 7ðG&7ð &7ð &7ñ &7ô &7ð &7ð &7ð &7ð &7ð &7ð &7øøøð &7ð &7ð &7ð &7øøõP Ôð ð ð õ „lÐ'Ñ(Ô(Ð(Ø€l€l€lð	øøøðW 	ñ /ðb 
Ðså   ÃA/NÄ3B/IÇ"4H"ÈIÈ"H&È&IÈ)H&È*&IÉNÉI 	É NÉ#I 	É$A8NËBNÍ5NÎN	ÎNÎN	Î	NÎNÎNÏW Ï4CWÓA$T4Ô3WÔ4U"ÕUÕWÕU"Õ"AW×W ×W×W ×W×W ×%XØ X)4Ú__doc__Ú
__future__r   r   r   rm   r³   rX   Útf_slim.trainingr   Útensorflow.core.protobufr   Útensorflow.python.clientr   Útensorflow.python.frameworkr	   r
   r   Útensorflow.python.lib.ior   Útensorflow.python.opsr   r   r   r   r   r   Útensorflow.python.platformr   rB   Útensorflow.python.summaryr   Útensorflow.python.trainingr   Útf_optimizerr   r¢   r   r   r   Ú__all__r   r   r   Ú_USE_GLOBAL_STEPÚ	OptimizerÚGATE_OPr   r\   r   r�   r   r€   r/   r.   ú<module>rÝ      sç  ðð"hð hðR 'Ð &Ð &Ð &Ð &Ð &Ø Ð Ð Ð Ð Ð Ø %Ð %Ð %Ð %Ð %Ð %à 	€	€	€	Ø 
€
€
€
Ø €€€à %Ð %Ð %Ð %Ð %Ð %à /Ð /Ð /Ð /Ð /Ð /Ø -Ð -Ð -Ð -Ð -Ð -Ø 3Ð 3Ð 3Ð 3Ð 3Ð 3Ø .Ð .Ð .Ð .Ð .Ð .Ø +Ð +Ð +Ð +Ð +Ð +Ø ,Ð ,Ð ,Ð ,Ð ,Ð ,Ø +Ð +Ð +Ð +Ð +Ð +Ø *Ð *Ð *Ð *Ð *Ð *Ø 2Ð 2Ð 2Ð 2Ð 2Ð 2Ø ,Ð ,Ð ,Ð ,Ð ,Ð ,Ø *Ð *Ð *Ð *Ð *Ð *Ø +Ð +Ð +Ð +Ð +Ð +Ø <Ð <Ð <Ð <Ð <Ð <Ø -Ð -Ð -Ð -Ð -Ð -Ø @Ð @Ð @Ð @Ð @Ð @Ø 8Ð 8Ð 8Ð 8Ð 8Ð 8Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ø >Ð >Ð >Ð >Ð >Ð >Ø 4Ð 4Ð 4Ð 4Ð 4Ð 4ðð ð €ð ð  ð  ð,'#ð '#ð '#ðTð ð ð6 Ð ð
 !1Ø#Ø'+Ø'(Ø(-Ø#/Ô#9Ô#AØ'+Ø05Ø)-Ø#'ðB%ð B%ð B%ð B%ðJð ð ð7!ð 7!ð 7!ðt €ð
 #Ø(ØØØØØØØØØ$ØØØ!Ø!Ø%ØØØ ØØØØ"Ø#ð3Ið Ið Ið Ið Ið Ir/   