Loss jumps to 87.3365

The id_loss = 87.3365 usually means the softmax loss is computing -log(FLT_MIN) here, which implies the probability is smaller than FLT_MIN.However, this probability is computed as softmax(Wx), where W is initialized as zero matrix in the first iteration.

As I could not reproduce this situation on my machine, could you please kindly do me a favor by adding the following print function in caffe/src/caffe/layers/softmax_loss_layer.cu

template <typename Dtype>
void debug_print(const Blob<Dtype>& prob) {
  printf("prob shape %d x %d\n", prob.shape(0), prob.shape(1));

  Dtype minval = (Dtype)FLT_MAX;
  Dtype maxval = -(Dtype)FLT_MAX;
  int minidx = 0;
  int maxidx = 0;

  const Dtype* data = prob.cpu_data();
  for (int i = 0; i < prob.count(); ++i) {
    if (data[i] <= minval) {
      minval = data[i];
      minidx = i;
    }
    if (data[i] >= maxval) {
      maxval = data[i];
      maxidx = i;
    }
  }

  printf("prob min = %.6f, at index %d\n", minval, minidx);
  printf("prob max = %.6f, at index %d\n", maxval, maxidx);
}

as well as adding debug_print(prob_);

after this line.Then recompile the caffe, rerun the experiments, and check the output before the Iteration 0. Thank you very much in advance.