Make policy used only for initial guidance

[oscardssmith]

Currently policy has a large effect on search even when there are millions of nodes. This causes weird behaviour and can lead to significant oversights. To fix this, I'm proposing that rather than maximizing Q + c_puct*policy*sqrt(N / n_i), we instead maximize Q + f(policy, N) + c_puct*sqrt(N/n_i). I don't have a great idea of what f should be, but a reasonable first aproximation is e^-(aN+policy), see the link for the graph. https://www.desmos.com/calculator/g3kmkjp5zs

[gcp]

I like this idea (read: I'm experimenting with it as well), it seems so counter-intuitive that policy is still a direct factor in the allocation of search effort even if we've searched thousands of nodes below a tree and have win-rates (which are better guides) for the subtrees.

[oscardssmith]

Yeah also, this detail of policy effecting U was invented by Deepmind with no justification.

[roy7]

I've never liked heavy weighting of policy in search even when millions of payouts have been done. I like to think of it more as a tool to guide very early search or know which unknown children to expand first.

Eager to hear about the results.

[gcp]

Yeah also, this detail of policy effecting U was invented by Deepmind with no justification.

The original paper about P-UCT had the prior as an additive, not a multiplicative factor. But I'm sure DM was well aware of that :-/

[roy7]

Maybe this is a case of them trying an experiment that led to a strength gain and so they kept it without a theoretical basis to do so.

[RedDenver]

This is from discord and didn't want to lose it:

the U computation comes from several places, here's how cpuct is computed:

inline float ComputeCpuct(const SearchParams& params, uint32_t N) {
  const float init = params.GetCpuct();
  const float k = params.GetCpuctFactor();
  const float base = params.GetCpuctBase();
  return init + (k ? k * FastLog((N + base) / base) : 0.0f);
}

then

float puct_mult =
        cpuct * std::sqrt(std::max(node->GetChildrenVisits(), 1u));

then

float GetU(float numerator) const {
    return numerator * GetP() / (1 + GetNStarted());
  }

where numerator=puct_mult

[kmcrage]

I agree about not maximizing
Q + c_puct * policy * sqrt(N / n_i)
but instead of adding an extra term, i though we could maximize
Q + c_puct * (1. - pow(1. - policy, 1. + c_pol * N)) * sqrt(N / n_i)
where c_pol is a small constant. Then as N increases the effect of policy dies out.
I'd also look at clamping policy at something slightly more than zero.

[oscardssmith]

I think the clamping isn't necessary. Search should be able to handle 0 policy without breaking.

[gcp]

I tried various experiments with pow(policy, softmax + increasing_term) to flatten out the policy as the number of playouts increases, but nothing that actually improves strenght so far.

[oscardssmith]

@gcp your problem might be that the change you described brings all policies to 0 instead of 1.

[gcp]

I was tuning the PUCT to compensate at the same time. But maybe it's a point that the policy distribution has to be renormalized.

[gcp]

@kmcrage I tried your formula, but I used x + y * sqrt(N) as the exponent in the pow. This seems to be worth about 40 Elo, but note that I am testing this without the DM formula that was increasing PUCT.

In any case this confirms the idea is worth exploring further.