Foraging

This notebook gathers different kinds of agents for foraging and target search in various scenarios, adapted for their use in the reinforcement learning paradigm.

Libs

Helpers

source

rand_choice_nb

def rand_choice_nb(
    arr:<built-in function array>, # A 1D numpy array of values to sample from.
    prob:<built-in function array>, # A 1D numpy array of probabilities for the given samples.
)->float: # A random sample from the given array with a given probability.

Forager

/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/fastcore/docments.py:478: UserWarning: docments unavailable for Forager: ValueError: The truth value of an empty array is ambiguous. Use `array.size > 0` to check that an array is not empty.
  warnings.warn(f'docments unavailable for {self.nm}: {type(e).__name__}: {e}')

source

Forager

Forager(num_actions: int, size_state_space: <built-in function array>, gamma_damping=0.0, eta_glow_damping=0.0, policy_type='standard', beta_softmax=3, initial_prob_distr=array([], shape=(2, 0), dtype=float64), fixed_policy=array([], shape=(2, 0), dtype=float64), max_no_H_update=10000, g_update='s')

This class defines a Forager agent, able to perform actions and learn from rewards based on the PS formalism.

This is an updated version from the one used in the original paper (https://doi.org/10.1088/1367-2630/ad19a8), taking into account the improvements made to the H and G matrices proposed by Michele Caraglio in our paper (https://doi.org/10.1039/D3SM01680C).

See self.act

Parallel training launchers

Here we gather useful functions that launch training and inference of agents for different environments

Reset Environment

Search loop

source

train_loop_reset

def train_loop_reset(
    episodes:int, # Number of episodes
    time_ep:int, # Length of an episode
    agent:object, # Agent that will be trained
    env:object, # Environment where the agent will be trained
    h_mat_allT:bool=False, # If True saves the h_mat at desired time
    when_save_h_mat:int=1, # If h_mat_allT = True, sets the time where h-matrix is saved
    reset_after_reward:bool=True, # If True, the agents performs a reset after getting a target
):

Training loop for a forager agent in a given environment.

Launchers

We now prepare some functions that allow us to launch parallel trainings with ease. We have to separate the launchers in 1D and 2D because of numba compilation, which would give errors due to the enviroments asking for different inputs.

1D

/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/fastcore/docments.py:478: UserWarning: docments unavailable for run_agents_reset_1D: ValueError: The truth value of an empty array is ambiguous. Use `array.size > 0` to check that an array is not empty.
  warnings.warn(f'docments unavailable for {self.nm}: {type(e).__name__}: {e}')

source

run_agents_reset_1D

run_agents_reset_1D(episodes, time_ep, N_agents, D=0.5, L=10.0, num_actions=2, size_state_space=array([100]), gamma_damping=1e-05, eta_glow_damping=0.1, g_update='s', initial_prob_distr=array([], shape=(2, 0), dtype=float64), policy_type='standard', beta_softmax=3, fixed_policy=array([], shape=(2, 0), dtype=float64), max_no_H_update=1000, h_mat_allT=False, when_save_h_mat=1, reset_after_reward=True, num_runs=None)

Launches parallel trainings of forager agents in a 1D Reset environment.

2D

/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/fastcore/docments.py:478: UserWarning: docments unavailable for run_agents_reset_2D: ValueError: The truth value of an empty array is ambiguous. Use `array.size > 0` to check that an array is not empty.
  warnings.warn(f'docments unavailable for {self.nm}: {type(e).__name__}: {e}')

source

run_agents_reset_2D

run_agents_reset_2D(episodes, time_ep, N_agents, dist_target=10.0, radius_target=1.0, D=0.5, num_actions=2, size_state_space=array([100]), gamma_damping=1e-05, eta_glow_damping=0.1, initial_prob_distr=array([], shape=(2, 0), dtype=float64), policy_type='standard', beta_softmax=3, fixed_policy=array([], shape=(2, 0), dtype=float64), max_no_H_update=1000, h_mat_allT=False, when_save_h_mat=1, reset_after_reward=True, g_update='s', num_runs=None)

Launches parallel trainings of forager agents in a 2D Reset environment.

nbdev