Trading and Reinforcement Learning
Stock trading is a natural application of reinforcement learning. The structure where a trader (agent) decides to buy/sell/hold (actions) in the market (environment) and earns profit (reward) matches the RL framework exactly.
Disclaimer
This article is an educational exercise for learning reinforcement learning, and it is not investment advice. Real financial markets are far more complex than what is covered here.
What we build here is not a system that makes money in the market but a practice exercise in defining an MDP yourself and implementing a custom Gymnasium environment from scratch. The price data used later on is not real quotes either -- it is synthetic data generated by code. So no number that comes out of this environment tells you anything about real markets. That is also why no performance figures appear anywhere in this article.
There is exactly one reason trading was chosen as the example. Among problems where you have to decide the state, the action, the reward, and the episode boundaries entirely on your own, it is one of the easier ones for grasping intuitively what each element corresponds to. CartPole and Atari in the earlier articles come with the environment already given, which lets you skip this design process. The skills you pick up here carry over directly to problems like inventory management and resource scheduling.
Trading Basics
Basic Terminology
- Buy/Long: Purchase stocks betting on price increase
- Sell/Short: Sell held stocks to close a position
- Position: Current state of held stocks
- Return: Ratio of profit to investment
- Commission: Cost incurred during transactions
- Slippage: Difference between order price and actual execution price
Modeling Trading as Reinforcement Learning
| RL Element | Trading Counterpart |
|---|---|
| State | Historical price data, technical indicators, current position |
| Action | Buy, Sell, Hold |
| Reward | Realized profit, unrealized P&L change |
| Episode | A trading session over a given period |
Data Preparation
Price Data Generation
We generate synthetic data for practice. For real applications, data can be fetched from Yahoo Finance, etc.
import numpy as np
import pandas as pd
def generate_stock_data(n_days=1000, initial_price=100.0, volatility=0.02, seed=42):
"""합성 주가 데이터 생성 (기하 브라운 운동 모델)"""
np.random.seed(seed)
daily_returns = np.random.normal(0.0005, volatility, n_days)
prices = initial_price * np.cumprod(1 + daily_returns)
data = pd.DataFrame()
data['close'] = prices
data['open'] = prices * (1 + np.random.normal(0, 0.005, n_days))
data['high'] = np.maximum(data['open'], data['close']) * (1 + np.abs(np.random.normal(0, 0.01, n_days)))
data['low'] = np.minimum(data['open'], data['close']) * (1 - np.abs(np.random.normal(0, 0.01, n_days)))
data['volume'] = np.random.randint(100000, 1000000, n_days).astype(float)
return data
stock_data = generate_stock_data(n_days=2000)
print(f"데이터 크기: {len(stock_data)}")
print(stock_data.head())
Technical Indicators
def add_technical_indicators(df):
"""기술적 지표 추가"""
df['sma_10'] = df['close'].rolling(window=10).mean()
df['sma_30'] = df['close'].rolling(window=30).mean()
delta = df['close'].diff()
gain = delta.where(delta > 0, 0).rolling(window=14).mean()
loss = (-delta.where(delta < 0, 0)).rolling(window=14).mean()
rs = gain / (loss + 1e-10)
df['rsi'] = 100 - (100 / (1 + rs))
bb_mean = df['close'].rolling(window=20).mean()
bb_std = df['close'].rolling(window=20).std()
df['bb_upper'] = bb_mean + 2 * bb_std
df['bb_lower'] = bb_mean - 2 * bb_std
df['bb_position'] = (df['close'] - df['bb_lower']) / (df['bb_upper'] - df['bb_lower'] + 1e-10)
ema12 = df['close'].ewm(span=12).mean()
ema26 = df['close'].ewm(span=26).mean()
df['macd'] = ema12 - ema26
df['macd_signal'] = df['macd'].ewm(span=9).mean()
df['returns'] = df['close'].pct_change()
df['returns_5d'] = df['close'].pct_change(5)
df.dropna(inplace=True)
df.reset_index(drop=True, inplace=True)
return df
stock_data = add_technical_indicators(stock_data)
print(f"지표 추가 후 데이터 크기: {len(stock_data)}")
print(f"특성 목록: {list(stock_data.columns)}")
Trading Environment Design
We implement a custom trading environment following the Gymnasium interface.
import gymnasium as gym
from gymnasium import spaces
class StockTradingEnv(gym.Env):
"""주식 트레이딩 환경"""
metadata = {"render_modes": ["human"]}
def __init__(self, df, window_size=30, commission=0.001,
initial_balance=100000):
super().__init__()
self.df = df
self.window_size = window_size
self.commission = commission
self.initial_balance = initial_balance
self.feature_columns = [
'close', 'volume', 'sma_10', 'sma_30', 'rsi',
'bb_position', 'macd', 'macd_signal', 'returns', 'returns_5d'
]
self.n_features = len(self.feature_columns)
self.action_space = spaces.Discrete(3)
obs_shape = self.window_size * self.n_features + 3
self.observation_space = spaces.Box(
low=-np.inf, high=np.inf, shape=(obs_shape,), dtype=np.float32
)
def _get_observation(self):
"""현재 관찰값 생성"""
start = self.current_step - self.window_size
end = self.current_step
window_data = self.df[self.feature_columns].iloc[start:end].values
for i in range(self.n_features):
col = window_data[:, i]
min_val = col.min()
max_val = col.max()
if max_val - min_val > 0:
window_data[:, i] = (col - min_val) / (max_val - min_val)
else:
window_data[:, i] = 0.0
flat_window = window_data.flatten()
position_info = np.array([
1.0 if self.position > 0 else 0.0,
self.unrealized_pnl / self.initial_balance,
self.shares * self.current_price / self.total_value,
], dtype=np.float32)
return np.concatenate([flat_window, position_info]).astype(np.float32)
@property
def current_price(self):
return self.df['close'].iloc[self.current_step]
@property
def unrealized_pnl(self):
if self.position > 0:
return self.shares * (self.current_price - self.entry_price)
return 0.0
@property
def total_value(self):
return self.balance + self.shares * self.current_price
def reset(self, seed=None, options=None):
super().reset(seed=seed)
self.current_step = self.window_size
self.balance = self.initial_balance
self.shares = 0
self.position = 0
self.entry_price = 0.0
self.total_trades = 0
self.winning_trades = 0
self.trade_history = []
return self._get_observation(), {}
def step(self, action):
prev_total = self.total_value
reward = 0.0
trade_info = ""
current_price = self.current_price
if action == 1 and self.position == 0:
max_shares = int(self.balance * 0.95 / (current_price * (1 + self.commission)))
if max_shares > 0:
cost = max_shares * current_price * (1 + self.commission)
self.balance -= cost
self.shares = max_shares
self.position = 1
self.entry_price = current_price
trade_info = f"매수 {max_shares}주 @ {current_price:.2f}"
elif action == 2 and self.position == 1:
proceeds = self.shares * current_price * (1 - self.commission)
self.balance += proceeds
pnl = (current_price - self.entry_price) / self.entry_price
self.total_trades += 1
if pnl > 0:
self.winning_trades += 1
self.trade_history.append(pnl)
trade_info = f"매도 {self.shares}주 @ {current_price:.2f}, 수익률: {pnl:.2%}"
self.shares = 0
self.position = 0
self.entry_price = 0.0
self.current_step += 1
current_total = self.total_value
reward = (current_total - prev_total) / prev_total
terminated = self.current_step >= len(self.df) - 1
truncated = self.total_value < self.initial_balance * 0.5
info = {
"total_value": self.total_value,
"balance": self.balance,
"position": self.position,
"total_trades": self.total_trades,
"trade_info": trade_info,
}
return self._get_observation(), reward, terminated, truncated, info
env = StockTradingEnv(stock_data, window_size=30)
obs, info = env.reset()
print(f"관찰 차원: {obs.shape}")
print(f"초기 포트폴리오: {env.total_value:,.0f}원")
Random Agent Baseline
def evaluate_random_agent(env, n_episodes=10):
"""무작위 에이전트 평가"""
results = []
for episode in range(n_episodes):
obs, _ = env.reset()
while True:
action = env.action_space.sample()
obs, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
break
final_value = info['total_value']
total_return = (final_value - env.initial_balance) / env.initial_balance
results.append({'final_value': final_value, 'return': total_return, 'trades': info['total_trades']})
returns = [r['return'] for r in results]
print(f"=== 무작위 에이전트 ({n_episodes}회) ===")
print(f"평균 수익률: {np.mean(returns):.2%}")
print(f"최대 수익률: {np.max(returns):.2%}")
print(f"최소 수익률: {np.min(returns):.2%}")
return results
# random_results = evaluate_random_agent(env)
Feedforward DQN Model
import torch
import torch.nn as nn
import torch.optim as optim
from collections import deque
import random
class TradingDQN(nn.Module):
"""트레이딩용 피드포워드 DQN"""
def __init__(self, obs_size, n_actions):
super().__init__()
self.net = nn.Sequential(
nn.Linear(obs_size, 256), nn.ReLU(), nn.Dropout(0.2),
nn.Linear(256, 128), nn.ReLU(), nn.Dropout(0.2),
nn.Linear(128, 64), nn.ReLU(),
nn.Linear(64, n_actions),
)
def forward(self, x):
return self.net(x)
CNN Model: Processing Price Charts Like Images
Temporal patterns in price data are captured using 1D convolutions.
class TradingCNN(nn.Module):
"""1D CNN 기반 트레이딩 모델"""
def __init__(self, window_size, n_features, n_actions):
super().__init__()
self.conv = nn.Sequential(
nn.Conv1d(n_features, 32, kernel_size=5, padding=2), nn.ReLU(),
nn.Conv1d(32, 64, kernel_size=3, padding=1), nn.ReLU(),
nn.AdaptiveAvgPool1d(1),
)
self.fc = nn.Sequential(
nn.Linear(64 + 3, 64), nn.ReLU(),
nn.Linear(64, n_actions),
)
self.window_size = window_size
self.n_features = n_features
def forward(self, x):
batch_size = x.shape[0]
window_data = x[:, :-3].view(batch_size, self.window_size, self.n_features)
position_info = x[:, -3:]
window_data = window_data.transpose(1, 2)
conv_out = self.conv(window_data).squeeze(-1)
combined = torch.cat([conv_out, position_info], dim=1)
return self.fc(combined)
Training the Trading Agent
class ReplayBuffer:
def __init__(self, capacity):
self.buffer = deque(maxlen=capacity)
def push(self, state, action, reward, next_state, done):
self.buffer.append((state, action, reward, next_state, done))
def sample(self, batch_size):
batch = random.sample(self.buffer, batch_size)
s, a, r, ns, d = zip(*batch)
return (np.array(s), np.array(a), np.array(r, dtype=np.float32),
np.array(ns), np.array(d, dtype=np.bool_))
def __len__(self):
return len(self.buffer)
def train_trading_agent(env, model_type="ff", n_episodes=500):
"""트레이딩 에이전트 학습"""
obs_size = env.observation_space.shape[0]
n_actions = env.action_space.n
device = torch.device("cpu")
if model_type == "cnn":
online_net = TradingCNN(env.window_size, env.n_features, n_actions).to(device)
target_net = TradingCNN(env.window_size, env.n_features, n_actions).to(device)
else:
online_net = TradingDQN(obs_size, n_actions).to(device)
target_net = TradingDQN(obs_size, n_actions).to(device)
target_net.load_state_dict(online_net.state_dict())
optimizer = optim.Adam(online_net.parameters(), lr=1e-4)
buffer = ReplayBuffer(50000)
epsilon = 1.0
epsilon_min = 0.05
epsilon_decay = 0.995
gamma = 0.99
batch_size = 64
target_update = 50
best_return = -float('inf')
returns_history = []
for episode in range(n_episodes):
obs, _ = env.reset()
total_reward = 0
while True:
if random.random() < epsilon:
action = env.action_space.sample()
else:
with torch.no_grad():
q = online_net(torch.tensor([obs], dtype=torch.float32).to(device))
action = q.argmax(dim=1).item()
next_obs, reward, terminated, truncated, info = env.step(action)
done = terminated or truncated
buffer.push(obs, action, reward, next_obs, done)
total_reward += reward
obs = next_obs
if len(buffer) >= batch_size:
s, a, r, ns, d = buffer.sample(batch_size)
s_t = torch.tensor(s, dtype=torch.float32).to(device)
a_t = torch.tensor(a, dtype=torch.long).to(device)
r_t = torch.tensor(r, dtype=torch.float32).to(device)
ns_t = torch.tensor(ns, dtype=torch.float32).to(device)
d_t = torch.tensor(d, dtype=torch.bool).to(device)
current_q = online_net(s_t).gather(1, a_t.unsqueeze(1)).squeeze(1)
with torch.no_grad():
best_a = online_net(ns_t).argmax(dim=1)
next_q = target_net(ns_t).gather(1, best_a.unsqueeze(1)).squeeze(1)
next_q[d_t] = 0.0
target_q = r_t + gamma * next_q
loss = nn.SmoothL1Loss()(current_q, target_q)
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(online_net.parameters(), 1.0)
optimizer.step()
if done:
break
if episode % target_update == 0:
target_net.load_state_dict(online_net.state_dict())
epsilon = max(epsilon_min, epsilon * epsilon_decay)
episode_return = (env.total_value - env.initial_balance) / env.initial_balance
returns_history.append(episode_return)
if episode_return > best_return:
best_return = episode_return
torch.save(online_net.state_dict(), "best_trading_model.pth")
if episode % 50 == 0:
mean_return = np.mean(returns_history[-50:])
print(f"에피소드 {episode}: 수익률={episode_return:.2%}, 평균 수익률={mean_return:.2%}, 거래={info['total_trades']}, 엡실론={epsilon:.3f}")
return online_net, returns_history
# trained_model, history = train_trading_agent(env, model_type="ff", n_episodes=500)
Agent Evaluation and Analysis
def backtest_agent(env, net, n_episodes=5):
"""학습된 에이전트 백테스트"""
all_results = []
for episode in range(n_episodes):
obs, _ = env.reset()
portfolio_values = [env.total_value]
while True:
with torch.no_grad():
q = net(torch.tensor([obs], dtype=torch.float32))
action = q.argmax(dim=1).item()
obs, reward, terminated, truncated, info = env.step(action)
portfolio_values.append(env.total_value)
if terminated or truncated:
break
total_return = (portfolio_values[-1] - portfolio_values[0]) / portfolio_values[0]
daily_returns = np.diff(portfolio_values) / portfolio_values[:-1]
sharpe_ratio = np.mean(daily_returns) / (np.std(daily_returns) + 1e-10) * np.sqrt(252)
peak = np.maximum.accumulate(portfolio_values)
drawdown = (np.array(portfolio_values) - peak) / peak
max_drawdown = drawdown.min()
result = {'total_return': total_return, 'sharpe_ratio': sharpe_ratio, 'max_drawdown': max_drawdown, 'total_trades': info['total_trades']}
all_results.append(result)
print(f"\n에피소드 {episode + 1}: 총 수익률: {total_return:.2%}, 샤프 비율: {sharpe_ratio:.2f}, 최대 낙폭: {max_drawdown:.2%}")
avg_return = np.mean([r['total_return'] for r in all_results])
print(f"\n=== 전체 백테스트 결과 === 평균 수익률: {avg_return:.2%}")
return all_results
# backtest_results = backtest_agent(env, trained_model)
Comparison with a Buy-and-Hold Strategy
def compare_with_buy_and_hold(df, agent_results, initial_balance=100000):
"""바이앤홀드 전략과 RL 에이전트 비교"""
# 바이앤홀드: 처음에 매수하고 끝까지 보유
start_price = df['close'].iloc[30] # window_size 이후
end_price = df['close'].iloc[-1]
bnh_return = (end_price - start_price) / start_price
agent_return = np.mean([r['total_return'] for r in agent_results])
print("=== 전략 비교 ===")
print(f"바이앤홀드 수익률: {bnh_return:.2%}")
print(f"RL 에이전트 수익률: {agent_return:.2%}")
print(f"초과 수익률: {agent_return - bnh_return:.2%}")
# compare_with_buy_and_hold(stock_data, backtest_results)
Why Reward Function Design Matters
The design of the reward function is decisive for the behaviour of a trading agent.
class ImprovedRewardEnv(StockTradingEnv):
"""개선된 보상 함수를 사용하는 트레이딩 환경"""
def _compute_reward(self, prev_total, action):
"""다양한 보상 설계 방식"""
current_total = self.total_value
# 1. 단순 포트폴리오 변화율
basic_reward = (current_total - prev_total) / prev_total
# 2. 리스크 조정 보상 (샤프 비율 유사)
# 수익률에서 변동성 페널티를 차감
volatility_penalty = 0.0
if len(self.trade_history) > 1:
recent_returns = self.trade_history[-10:]
volatility_penalty = np.std(recent_returns) * 0.1
# 3. 과도한 거래 페널티
trade_penalty = 0.0
if action != 0: # 보유가 아닌 경우
trade_penalty = -0.0001
# 4. 승률 보너스
win_bonus = 0.0
if self.total_trades > 10:
win_rate = self.winning_trades / self.total_trades
if win_rate > 0.5:
win_bonus = 0.0001
return basic_reward - volatility_penalty + trade_penalty + win_bonus
What You Actually See When You Run the Training
Run train_trading_agent as it stands and one line of log piles up per episode. What matters here is not the return number but whether there is any sign that learning is progressing at all. Telling that apart takes looking at three things together.
The first is epsilon. epsilon_decay in the code is 0.995, so it shrinks by half a percent each episode. Starting from 1.0, reaching epsilon_min of 0.05 takes roughly 600 episodes. But the default n_episodes is 500. In other words, even running this configuration all the way through leaves the agent acting randomly with a probability of a little over 6 percent right up to the final moment. If the metrics late in training look erratic, it may not be that the agent failed to learn but that it is still exploring. Evaluation has to be done separately, with epsilon switched off. That is exactly why backtest_agent uses nothing but argmax.
The second is the trade count. Print info['total_trades'] on every episode. Whether that value converges toward zero or runs away to the point of trading at every single step is the fastest signal you have of whether the reward function is doing its job. Both are common failure modes, and both are covered separately below.
The third is the loss. The TD loss in DQN does not come down smoothly the way supervised learning does. The target network is refreshed every target_update episodes, so a spike in the loss at each of those points is normal. The patterns that indicate a real problem are different ones: the loss diverging, or, conversely, sticking to zero very early on. The latter usually means the network is emitting the same Q value for every state -- in other words, it has collapsed.
Patching the training loop a little so all three land in the log makes diagnosis much easier.
# train_trading_agent의 에피소드 루프 끝에 추가
if episode % 10 == 0:
print(
f"ep={episode:4d} "
f"eps={epsilon:.3f} "
f"trades={env.total_trades:3d} "
f"value={env.total_value:,.0f} "
f"loss={np.mean(recent_losses) if recent_losses else float('nan'):.5f}"
)
recent_losses = []
The output ends up looking roughly like this. Read how the columns move together rather than the numbers themselves.
ep= 0 eps=1.000 trades= 87 value=... loss=0.01243
ep= 10 eps=0.951 trades= 74 value=... loss=0.00871
ep= 20 eps=0.905 trades= 61 value=... loss=0.00655
...
ep= 200 eps=0.367 trades= 9 value=... loss=0.00112
ep= 400 eps=0.135 trades= 2 value=... loss=0.00038
A pattern where the trade count falls monotonically in step with epsilon is not a good sign. The agent has not learned a strategy; trading dropped by exactly as much as the random actions did. Which is to say the learned policy is effectively hold-only, and every trade left over is exploration noise. The first item in the next section is precisely this situation.
Failure Cases and a Diagnostic Order
The agent does nothing. This is the most common ending. The symptom is total_trades sitting at zero or in the single digits while the portfolio value barely moves from where it started. The cause is in the reward structure. commission is 0.001, so one buy plus one sell makes a round trip that costs 0.2 percent as a guaranteed loss. Holding, by contrast, costs nothing. Faced with a certain loss and an uncertain gain, Q-learning choosing the certain side is perfectly rational behaviour. The diagnostic order goes like this. First set commission=0.0 and train again. If trading comes back to life, that pins the cause on the scale of the reward relative to the commission. Then restore the commission and rebalance from the other end, either by scaling the reward up or by giving the holding state a very small time penalty. Leaving the commission removed is not an answer. That does not solve the problem, it deletes it.
Conversely, it trades at every step. This is the opposite failure, easy to fall into while scaling the reward up. It is why ImprovedRewardEnv carries a trade_penalty at all. Hand-tuning the penalty constant is closer to a stopgap, though. The more robust move is to change the action space itself. Make the action a target position weight instead of allowing a buy and a sell at every step, and the action that keeps the same weight naturally becomes no trade, so overtrading disappears without any penalty at all.
The same result repeats five times. Run backtest_agent(env, net, n_episodes=5) and the metrics for all five episodes come out completely identical. It looks like a bug, but it is exactly what the code says. reset() always sets current_step back to window_size, df is the same every time, and the evaluation policy is argmax, which is deterministic. A deterministic policy in a deterministic environment has exactly one trajectory. In other words, the sample size of this backtest is not five, it is one. Computing a mean and a standard deviation over that is meaningless. Doing it properly means picking the starting point at random in reset(), or evaluating against several different df objects covering different periods.
It only does well on the training data. This is the more fundamental issue that pairs with the one above. The current training loop walks the same single price path over and over for all 500 episodes. In reinforcement learning that is the equivalent of training on one training example. What the agent learns is not a generalized strategy but memorization of what it should have done at specific points on that specific path. When you split the data, always cut it in time order. Splitting a time series by shuffling it randomly creates a situation where you train on future segments and evaluate on past ones, and that result means nothing at all.
The indicators were computed over the whole range in advance. This is a quiet and dangerous trap. Indicators like sma_30 and rsi are safe when computed with pandas rolling, because they only ever look backwards. But the moment you take your normalization statistics as the mean and standard deviation of the entire dataframe, future information leaks into the observations at training time. The _get_observation in this article normalizes using only the minimum and maximum inside the window, and that is a design choice made to avoid exactly this. Be careful not to break that property when you rework the preprocessing. The symptom shows up as good metrics during training that fall apart on a new segment, and it is famously hard to trace back to its cause.
The Sharpe ratio comes out at a strange value. The Sharpe calculation in backtest_agent multiplies by np.sqrt(252). That constant is an annualization factor that assumes daily bars. Feed in hourly or minute data while leaving the value untouched and the result becomes meaningless. On top of that, when there is almost no trading and the daily returns are mostly zero, the denominator approaches zero and the value runs away. The 1e-10 in the code only prevents a division by zero; it does not preserve the meaning of the number. Always look at total_trades first, before you read any metric.
When Not to Use This Approach
Examples that handle trading with reinforcement learning turn up often in textbooks, but whether this problem structure actually suits reinforcement learning is worth examining on its own.
The conditions under which reinforcement learning does well are typically these. The environment is broadly stable, a simulator can cheaply generate an unlimited amount of experience, actions genuinely affect the next state, and the reward is comparatively clear. Games and robot control fit this precisely.
Market data satisfies almost none of these conditions. The rules themselves change over time, so there is no guarantee that what was learned from the past holds in the future. Experience is not cheap. Real data is finite, and padding it out with synthetic data means learning the assumptions of that synthesis process. The signal-to-noise ratio is extreme in the other direction as well, so the sample size needed to distinguish a result earned by a good policy from one earned by luck is beyond what anyone can afford.
So the honest way to order things is this. If it is a prediction problem, solve it with supervised learning first. Guessing the direction or the volatility of the next segment is something you can handle without reinforcement learning, and it is far easier to validate. Where reinforcement learning contributes uniquely is not prediction but sequential decision-making -- position sizing, order execution, and the like. Stack reinforcement learning on top before predictive performance is secured and, when things go badly, you will not be able to tell whether the cause is the prediction, the policy, or the reward design.
One more thing. The code in this article is deliberately simplified for teaching purposes. There is no slippage, orders always fill in full at the price you wanted, there is no market impact, and there is no intraday price movement. Introducing slippage in the terminology section and then leaving it out of the environment is one example of this. Undoing these simplifications one at a time makes a good follow-up exercise, and feeling how much harder the problem gets with each one you undo is the real payoff of this practice.
References
- Gymnasium: Making a Custom Environment (gymnasium.farama.org) -- where the return contracts of
resetandstepand the wayobservation_spaceandaction_spaceare defined were confirmed. Checked 2026-08-16. - Gymnasium Env API (gymnasium.farama.org) -- the definition of the five values
stepreturns and the distinction betweenterminatedandtruncated. Checked 2026-08-16. - Human-level control through deep reinforcement learning (Nature, 2015) -- the original DQN paper, which introduced the experience replay buffer and the target network.
- Earlier articles in this series covered the components of DQN. This one focused on the part where that structure is moved into a custom environment.
Summary
- Problem definition: Model stock trading as an MDP (state=market data, action=buy/sell/hold, reward=profit)
- Data preparation: Add technical indicators to price data to construct observation space
- Environment design: Implement custom trading environment with Gymnasium interface
- Models: Both feedforward DQN and 1D CNN models can be used
- Reward design: Consider various factors beyond simple returns, such as risk adjustment and trading penalties
- Evaluation: Evaluate with diverse performance metrics including returns, Sharpe ratio, and maximum drawdown
In the next article, we will move beyond value-based methods to explore Policy Gradient methods that directly optimize the policy.