Adaptive Hybrid Particle Swarm Optimization with Gradient Descent

# Adaptive Hybrid Particle Swarm Optimization with Gradient Descent
Source: [https://arxiv.org/html/2608.11258](https://arxiv.org/html/2608.11258)
###### Abstract

Gradient injection helps Particle Swarm Optimization \(PSO\) only when the swarm has identified a basin with smooth local structure—not universally\. We propose Adaptive Hybrid PSO \(AHPSO\), which uses a sigmoid function on swarm diversity to automatically modulate gradient influence: near\-zero during exploration, near\-maximum during exploitation, with no manual phase\-switching\. Under budget\-normalized comparison \(PSO given equivalent total function evaluations\), PSO wins 52\.5% of 40 configurations versus AHPSO’s 20% \(p=7\.0×10−5p=7\.0\\times 10^\{\-5\}, Friedman\)\. AHPSO retains advantage specifically on problems with smooth local basins \(F8, F24–F27\) where directed descent outperforms undirected sampling even at equal cost\. Under iteration\-matched comparison across 29 functions \(42 configurations, 14,700 runs\), AHPSO\-Adadelta ranks first of 9 methods including CMA\-ES \(p=9\.75×10−4p=9\.75\\times 10^\{\-4\}\)\. The contribution is a principled characterization ofwhengradient injection provides value in swarm\-based search, not a claim of universal superiority\.

## IBackground and Related Work

### I\-AWhat is Optimization?

Many real\-world problems require finding the best solution from a large set of possibilities, this isoptimization\. For example, finding the lowest point in a landscape \(Fig\.[1](https://arxiv.org/html/2608.11258#S1.F1)\)\. Simple landscapes have one valley \(unimodal\), but real problems often have many valleys \(multimodal\), making it easy to get stuck in a suboptimal solution\.

![Refer to caption](https://arxiv.org/html/2608.11258v1/x1.png)Figure 1:Two types of optimization landscapes\. \(a\) Unimodal: one clear optimum, easy to solve\. \(b\) Multimodal: many local optima that can trap algorithms, much harder\.
### I\-BParticle Swarm Optimization \(PSO\)

PSO\[[1](https://arxiv.org/html/2608.11258#bib.bib1)\]is a population\-based algorithm inspired by bird flocking\. A swarm ofNNparticles flies through add\-dimensional search space, each remembering its own best position \(pip\_\{i\}\) and knowing the swarm’s best position \(gg\)\. At each step, particleii’s velocity is updated using three forces \(Fig\.[2](https://arxiv.org/html/2608.11258#S1.F2)\):

vit\+1=w⋅vit⏟inertia\+c1​r1​\(pi−xit\)⏟cognitive\+c2​r2​\(g−xit\)⏟socialv\_\{i\}^\{t\+1\}=\\underbrace\{w\\cdot v\_\{i\}^\{t\}\}\_\{\\text\{inertia\}\}\+\\underbrace\{c\_\{1\}r\_\{1\}\(p\_\{i\}\-x\_\{i\}^\{t\}\)\}\_\{\\text\{cognitive\}\}\+\\underbrace\{c\_\{2\}r\_\{2\}\(g\-x\_\{i\}^\{t\}\)\}\_\{\\text\{social\}\}\(1\)xit\+1=xit\+vit\+1x\_\{i\}^\{t\+1\}=x\_\{i\}^\{t\}\+v\_\{i\}^\{t\+1\}\(2\)
Herewwis theinertia weight\(how much the particle trusts its current direction\),c1c\_\{1\}andc2c\_\{2\}are acceleration coefficients \(set to 2\.0 in our experiments\), andr1,r2∼U​\(0,1\)r\_\{1\},r\_\{2\}\\sim U\(0,1\)are fresh random numbers drawn each iteration, they inject stochasticity so particles don’t all follow the same path\.

![Refer to caption](https://arxiv.org/html/2608.11258v1/x2.png)Figure 2:PSO velocity update\. Each particle is pulled by three forces: inertia \(keep moving\), cognitive \(return to personal best\), and social \(move toward global best\)\. The combination determines the new position\.The inertia weightwwcontrols the balance between exploration and exploitation; Shi and Eberhart\[[9](https://arxiv.org/html/2608.11258#bib.bib9)\]showed that linearly decreasingwwfrom 0\.9 to 0\.4 over the run significantly improves convergence, highwwearly encourages broad search, lowwwlate encourages settling\.

Strengths:Simple, few parameters, good at exploring broadly\. Weakness:Slow to converge precisely once near the optimum\.

### I\-CGradient Descent \(GD\)

The gradient∇f​\(x\)\\nabla f\(x\)is a vector pointing in the direction of steepestascentat pointxx\. Gradient descent simply steps in the opposite direction, downhill:

xt\+1=xt−η⋅∇f​\(xt\)x^\{t\+1\}=x^\{t\}\-\\eta\\cdot\\nabla f\(x^\{t\}\)\(3\)
whereη\\etais the learning rate \(step size\)\. This is powerful because it useslocal shape information: instead of searching blindly, the algorithm knows exactly which direction improves the objective\. On a smooth bowl, this means exponentially fast convergence to the bottom\.

The fatal flaw is equally intuitive: the gradient only sees the local slope\. If the landscape has multiple valleys, GD rolls into whichever valley it starts in and stays there forever, it has no mechanism to “jump out” and explore other regions\.

Strengths:Very fast local convergence, precise\. Weakness:Gets trapped in the nearest local optimum; needs a good starting point\.

### I\-DThe Idea: Combine Both

PSO is good atexploration\(finding the right region\) but slow atexploitation\(refining the solution\)\. GD is the opposite\. Since no single algorithm dominates all problems\[[25](https://arxiv.org/html/2608.11258#bib.bib25)\], hybridization is appealing, but the central question iswhengradient injection helps, not whether it can\. Our budget\-normalized experiments \(Section[IV\-G](https://arxiv.org/html/2608.11258#S4.SS7)\) show that PSO with equivalent function evaluations wins 52\.5% of configurations, gradient direction provides value only on problems with smooth local basins where directed descent outperforms undirected sampling\. The mechanism we propose uses swarm diversity as a real\-time signal: gradients are informative only when the swarm has found a promising basin with smooth local structure; applied too early or on rugged landscapes, they add cost without benefit\.

### I\-ERelated Work

This timing question is not new, others have tried to combine PSO with local search\. Understanding what they did \(and what they left unsolved\) motivates our specific design choices\.

#### I\-E1PSO with Local Search

The earliest PSO\-gradient hybrid is Noel and Jannett\[[8](https://arxiv.org/html/2608.11258#bib.bib8)\], who applied gradient descent to the global best particle only\. This improved unimodal convergence but provided limited multimodal benefit because only one particle received gradient information\. Fan and Yan\[[16](https://arxiv.org/html/2608.11258#bib.bib16)\]extended this to a full hybrid PSO with local search applied to multiple particles, showing improvement on engineering design problems\. The memetic algorithm framework\[[14](https://arxiv.org/html/2608.11258#bib.bib14)\]formalizes this combination: evolutionary search provides global exploration while local refinement \(gradient\-based or otherwise\) accelerates exploitation\. Our work fits within this framework but adds an automatic mechanism for decidingwhento apply local refinement\.

#### I\-E2Adaptive PSO Variants

Rather than adding external operators, several approaches adapt PSO’s own parameters\. Zhan et al\.\[[10](https://arxiv.org/html/2608.11258#bib.bib10)\]proposed Adaptive PSO \(APSO\), which classifies the swarm into four evolutionary states using a fuzzy system on a diversity measure similar to ours, then adjustsww,c1c\_\{1\},c2c\_\{2\}accordingly\. Ratnaweera et al\.\[[15](https://arxiv.org/html/2608.11258#bib.bib15)\]introduced HPSO\-TVAC with time\-varying acceleration coefficients\. Liang et al\.\[[11](https://arxiv.org/html/2608.11258#bib.bib11)\]proposed Comprehensive Learning PSO \(CLPSO\), where each particle learns from different exemplars per dimension, this prevents premature convergence without external operators\. These methods address exploration\-exploitation balance through parameter adaptation alone; our approach instead modulates an external gradient operator, providing stronger exploitation than parameter tuning can achieve\.

#### I\-E3Hybrid Evolutionary\-Gradient Methods

Bosman and de Jong\[[23](https://arxiv.org/html/2608.11258#bib.bib23)\]combined gradient techniques with evolutionary multi\-objective optimization, demonstrating that gradient information accelerates convergence when available\. Epitropakis et al\.\[[22](https://arxiv.org/html/2608.11258#bib.bib22)\]hybridized PSO with Differential Evolution \(DE\), using DE’s mutation operator to enhance diversity, a complementary approach to our gradient\-based exploitation\. Lim and Isa\[[24](https://arxiv.org/html/2608.11258#bib.bib24)\]proposed a two\-layer PSO with intelligent division of labor between exploration and exploitation subswarms\.

#### I\-E4Alternative Optimization Paradigms

CMA\-ES\[[12](https://arxiv.org/html/2608.11258#bib.bib12)\]represents a fundamentally different approach: it maintains a covariance matrix that captures second\-order landscape information without explicit gradients, adapting the search distribution shape over iterations\. SHADE\[[13](https://arxiv.org/html/2608.11258#bib.bib13)\]and L\-SHADE use success\-history\-based parameter adaptation in Differential Evolution, achieving state\-of\-the\-art performance on CEC benchmarks\. Bonyadi and Michalewicz\[[21](https://arxiv.org/html/2608.11258#bib.bib21)\]survey the broader PSO landscape, identifying exploration\-exploitation balance as the central open challenge\. These methods serve as important baselines because they solve the exploration\-exploitation tradeoff through different mechanisms than gradient injection\.

#### I\-E5Positioning of Our Work

Our approach bridges adaptive PSO and memetic algorithms: like APSO, we use diversity to detect the swarm’s evolutionary state, but instead of adjusting PSO parameters, we modulate the strength of an external gradient operator\. The sigmoid gating function provides a principled, continuous transition, no thresholds to tune, no manual phase\-switching, that is self\-correcting: if diversity rebounds \(e\.g\., after a perturbation\), gradient influence automatically decreases\.

## IIProposed Method: AHPSO

The central challenge istiming: apply gradients too early and you kill exploration; apply them too late and you waste iterations\. We need a signal that tells us where the swarm is in its search process\. Our signal isdiversity, how spread out the particles are\. When particles are scattered, the swarm is still exploring and gradient descent would pull particles toward the nearest \(possibly wrong\) local optimum\. When particles cluster together, the swarm has found a promising region and gradient descent can safely refine the solution\.

### II\-ADiversity Measurement

Intuitively, we want a single number that captures “how spread out is the swarm?” The simplest robust choice is the average standard deviation of particle positions across each dimension\. This works because: \(1\) it is zero only when all particles occupy the same point \(full convergence\), \(2\) it scales naturally with the search space, and \(3\) it is cheap to compute\. For a swarm ofNNparticles indddimensions:

D​\(t\)=1d​∑j=1dσj​\(t\),σj​\(t\)=1N​∑i=1N\(xi,jt−x¯jt\)2D\(t\)=\\frac\{1\}\{d\}\\sum\_\{j=1\}^\{d\}\\sigma\_\{j\}\(t\),\\quad\\sigma\_\{j\}\(t\)=\\sqrt\{\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\bigl\(x\_\{i,j\}^\{t\}\-\\bar\{x\}\_\{j\}^\{t\}\\bigr\)^\{2\}\}\(4\)
wherex¯jt=1N​∑i=1Nxi,jt\\bar\{x\}\_\{j\}^\{t\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}x\_\{i,j\}^\{t\}is the swarm centroid along dimensionjj\. We use the population standard deviation \(dividing byNN, notN−1N\{\-\}1\) since we observe the entire swarm, not a sample from it\.

### II\-BAdaptive Gradient Weight

We need a function that maps diversity to gradient influence\. Three options:

- •Linear ramp, simple, but treats all diversity levels equally\. A drop from 90% to 80% increases gradient weight the same as a drop from 30% to 20%, even though only the latter signals real convergence\.
- •Step function, switches abruptly at a threshold, risking instability if diversity fluctuates near that point\.
- •Sigmoid, nearly flat at the extremes \(robust to noise\) and transitions smoothly in the middle \(responsive to genuine convergence\)\.

We choose the sigmoid\. Formally:

α​\(t\)=αmin\+\(1−αmin\)1\+ek​\(D​\(t\)D​\(0\)−τ\)\\alpha\(t\)=\\alpha\_\{\\min\}\+\\frac\{\(1\-\\alpha\_\{\\min\}\)\}\{1\+e^\{k\\left\(\\frac\{D\(t\)\}\{D\(0\)\}\-\\tau\\right\)\}\}\(5\)
The three parameters have intuitive interpretations:

- •αmin=0\.1\\alpha\_\{\\min\}=0\.1: the minimum gradient influence\. Even during exploration, a small gradient nudge helps particles descend within their local basin without disrupting the global search\.
- •τ=0\.3\\tau=0\.3: the sigmoid midpoint\. We want the transition to happenafterthe swarm has committed to a region butbeforeit has fully converged \(when gradients would add nothing\)\. Empirically, diversity drops below 30% of its initial value once particles cluster within a few basins, this is the sweet spot where gradient refinement becomes productive\.
- •k=5k=5: the steepness\. A moderate value ensures the transition spans roughly 20% of the diversity range \(fromα≈0\.2\\alpha\\approx 0\.2atD/D0=0\.4D/D\_\{0\}=0\.4toα≈0\.9\\alpha\\approx 0\.9atD/D0=0\.2D/D\_\{0\}=0\.2\), giving a smooth but decisive switch\.

![Refer to caption](https://arxiv.org/html/2608.11258v1/x3.png)Figure 3:The sigmoid function that controls gradient influence\. When particles are spread out \(high diversity ratio\),α\\alphais small, PSO dominates\. When particles converge \(low diversity\),α\\alphagrows, gradient descent takes over\.
### II\-CHow It Works in Practice

Fig\.[4](https://arxiv.org/html/2608.11258#S2.F4)shows a typical optimization run\. Early on, diversity is high andα≈0\.1\\alpha\\approx 0\.1\(mostly PSO\)\. As the swarm converges, diversity drops andα\\alpharises toward 1\.0 \(mostly GD\)\. The transition happens smoothly and automatically, no iteration counter or manual schedule is involved\.

What should you look for in these curves? The diversity curve is typically monotonically decreasing, but not always\. On multimodal functions, diversity can temporarilyincreaseif particles scatter after escaping a local optimum\. When this happens,α\\alphadrops back down automatically, the sigmoid acts as a safety valve, reducing gradient influence whenever the swarm re\-enters an exploratory phase\. This self\-correcting behavior is the key advantage over fixed schedules, which would continue applying strong gradients even when the swarm has not yet committed to a region\.

![Refer to caption](https://arxiv.org/html/2608.11258v1/x4.png)Figure 4:A typical run: diversity \(blue\) decays as particles converge, causing the gradient weightα\\alpha\(red\) to increase\. No manual tuning needed, the algorithm adapts itself\.
### II\-DThe Complete Algorithm

![Refer to caption](https://arxiv.org/html/2608.11258v1/x5.png)Figure 5:AHPSO pipeline showing the feedback loop that makes the method adaptive\. Steps \(1\), \(2\) are standard PSO; the novelty is steps \(3\), \(4\): diversity measurement feeds into the sigmoid \(Eq\.[5](https://arxiv.org/html/2608.11258#S2.E5)\), which outputsα\\alpha, the gradient scaling factor\. Early in the run, high diversity produces smallα\\alpha\(PSO dominates\)\. As particles converge,α\\alphagrows and gradient descent takes over refinement\. No manual phase\-switching is needed\.After the standard PSO update \(Eqs\.[1](https://arxiv.org/html/2608.11258#S1.E1)–[2](https://arxiv.org/html/2608.11258#S1.E2)\), each particle gets a gradient refinement step at every iterationtt:

xit\+1←xit\+1\+α​\(t\)⋅GD\_step​\(xit\+1\)x\_\{i\}^\{t\+1\}\\leftarrow x\_\{i\}^\{t\+1\}\+\\alpha\(t\)\\cdot\\text\{GD\\\_step\}\(x\_\{i\}^\{t\+1\}\)\(6\)
How do we get gradients?These benchmark functions are black boxes, we cannot compute analytical derivatives\. Instead, we estimate the gradient numerically using central differences:

∂f∂xj≈f​\(x\+ϵ​ej\)−f​\(x−ϵ​ej\)2​ϵ\\frac\{\\partial f\}\{\\partial x\_\{j\}\}\\approx\\frac\{f\(x\+\\epsilon\\,e\_\{j\}\)\-f\(x\-\\epsilon\\,e\_\{j\}\)\}\{2\\epsilon\}\(7\)
whereeje\_\{j\}is the unit vector along dimensionjjandϵ=10−8\\epsilon=10^\{\-8\}\. This requires2​d2dextra function evaluations per particle per iteration \(one forward and one backward perturbation per dimension\), a cost we quantify in Section[IV\-G](https://arxiv.org/html/2608.11258#S4.SS7)\.

The gradient step in Eq\.[6](https://arxiv.org/html/2608.11258#S2.E6)can use any first\-order optimizer\. We test six, spanning a spectrum from simple \(one fixed hyperparameter\) to fully self\-tuning \(no learning rate at all\)\. The key distinction is how each handles the learning rate problem: a rate that works on one landscape may diverge on another\.

- •SGD, one fixed step size for all dimensions\. Fast when tuned correctly, but a single bad learning rate causes divergence\.
- •Adagrad\[[5](https://arxiv.org/html/2608.11258#bib.bib5)\], accumulates past gradients per dimension, automatically shrinking the step for frequently\-updated directions\. Good for sparse problems but can stall as the accumulator grows\.
- •RMSprop\[[6](https://arxiv.org/html/2608.11258#bib.bib6)\], like Adagrad but uses an exponential moving average, preventing the step from shrinking to zero over time\.
- •Adam\[[3](https://arxiv.org/html/2608.11258#bib.bib3)\], combines per\-dimension adaptation \(like RMSprop\) with momentum \(remembering past directions\)\. The most popular optimizer in deep learning\.
- •Nadam\[[7](https://arxiv.org/html/2608.11258#bib.bib7)\], Adam with Nesterov lookahead: it evaluates the gradient at a predicted future position, giving slightly better convergence on smooth landscapes\.
- •Adadelta\[[4](https://arxiv.org/html/2608.11258#bib.bib4)\], eliminates the learning rate entirely by scaling updates using the ratio of past parameter changes to past gradients\. Fully self\-calibrating\.

Hyperparameters for reproducibility\.All optimizers use their standard defaults from the deep learning literature: Adam and Nadam useβ1=0\.9\\beta\_\{1\}\{=\}0\.9,β2=0\.999\\beta\_\{2\}\{=\}0\.999,ϵ=10−8\\epsilon\{=\}10^\{\-8\}; RMSprop uses decay rateρ=0\.9\\rho\{=\}0\.9,ϵ=10−8\\epsilon\{=\}10^\{\-8\}; Adagrad usesϵ=10−8\\epsilon\{=\}10^\{\-8\}; Adadelta usesρ=0\.95\\rho\{=\}0\.95,ϵ=10−6\\epsilon\{=\}10^\{\-6\}\. The base learning rateη\\eta\(Section[III](https://arxiv.org/html/2608.11258#S3)\) is 0\.01 for unimodal and 0\.001 for multimodal functions, a known limitation requiring problem\-class knowledge that we explicitly acknowledge \(see Section[III](https://arxiv.org/html/2608.11258#S3)\)\. Adadelta ignores this parameter entirely, which partly explains its robustness\.

## IIIExperimental Setup

The algorithm is defined, now we need to answer three questions: \(1\) Does the adaptive mechanism actually help, or does it just add overhead? \(2\) Which gradient optimizer pairs best with PSO? \(3\) On whichtypesof problems does hybridization help or hurt? To answer these, we need a diverse benchmark suite that separates easy problems \(where any method works\) from hard ones \(where exploration matters\)\.

### III\-ABenchmark Functions

We use 29 standard test functions\[[2](https://arxiv.org/html/2608.11258#bib.bib2)\], deliberately chosen to span a range of difficulties \(Table[I](https://arxiv.org/html/2608.11258#S3.T1)\)\.

TABLE I:Benchmark suite: 29 functions tested at multiple dimensionalities, yielding 42 total configurations \(F1–F13 tested at bothd=10d\{=\}10andd=30d\{=\}30; F14–F29 at fixed dimensions\)\.
### III\-BSettings

All methods use identical PSO parameters to ensure a fair comparison:

- •30 particles, 500 iterations, standard in the PSO literature\[[2](https://arxiv.org/html/2608.11258#bib.bib2)\]; enough particles for diversity measurement to be meaningful, enough iterations for convergence ond=30d\{=\}30problems\.
- •50 independent runs, sufficient for statistical tests \(Mann\-Whitney U requires≥\\geq20 samples for reliablepp\-values; 50 gives comfortable power\)\.
- •Inertiaww: 0\.9→\\to0\.4, the Shi, Eberhart schedule\[[9](https://arxiv.org/html/2608.11258#bib.bib9)\], widely adopted as a strong default\.
- •Acceleration coefficientsc1=c2=2\.0c\_\{1\}=c\_\{2\}=2\.0, the standard choice from Kennedy and Eberhart\[[1](https://arxiv.org/html/2608.11258#bib.bib1)\]that balances cognitive and social components\.
- •GD learning rateη\\eta: 0\.01 \(unimodal\), 0\.001 \(multimodal\), these are the base step sizes for the gradient descent component \(Eq\.[6](https://arxiv.org/html/2608.11258#S2.E6)\), not PSO parameters\. Unimodal landscapes have reliable gradients so larger steps are safe; multimodal landscapes need smaller steps to avoid overshooting into wrong basins\. Note: adaptive optimizers \(Adam, Adadelta, RMSprop\) internally rescale this rate per dimension, so the base value matters most for SGD\.Limitation:this two\-rate scheme requires knowing the problem class a priori, a form of oracle knowledge unavailable in practice\. This is a genuine weakness: results for SGD, Adagrad, and RMSprop benefit from this tuning and would degrade with a single universal rate\. Adadelta eliminates this issue entirely \(it ignoresη\\eta\), which partly explains its top ranking and makes it the recommended default for practitioners without problem\-class knowledge\.

### III\-CMethods Compared

We compare seven methods: standard PSO \(the baseline\) plus six AHPSO variants, one per gradient optimizer\. The experimental design isolates a single variable: all seven methods use identical PSO parameters, identical swarm size, and identical iteration budgets\. The six hybrids also share the same adaptive sigmoid mechanism with the sameτ\\tau,kk, andαmin\\alpha\_\{\\min\}\. Theonlydifference between them is which optimizer computes the gradient step in Eq\.[6](https://arxiv.org/html/2608.11258#S2.E6)\. This controlled setup means any performance difference must come from the gradient optimizer itself, not from tuning advantages or different exploration budgets\.

In total, this produces7×42×50=14,7007\\times 42\\times 50=14\{,\}700optimization runs\. We now present what they reveal\.

## IVResults

We address the three questions from Section[III](https://arxiv.org/html/2608.11258#S3)in order: first the overall ranking \(does hybridization help?\), then head\-to\-head comparisons \(which optimizer is best?\), and finally per\-group analysis \(where does it help or hurt?\)\.

### IV\-AOverall Ranking

Before examining individual functions, we need a single answer: across all 42 configurations, does the choice of method matter at all? The Friedman test\[[18](https://arxiv.org/html/2608.11258#bib.bib18)\], a non\-parametric alternative to repeated\-measures ANOVA suited to rank data, answers this\. It tests the null hypothesis that all seven methods perform identically; rejection means at least one method is reliably different\. Following the methodology of Derrac et al\.\[[20](https://arxiv.org/html/2608.11258#bib.bib20)\], we rank methods by mean performance per configuration\.

The result:χ2​\(6\)=14\.17\\chi^\{2\}\(6\)=14\.17,p=0\.028p=0\.028, significant at the 5% level but not at 1%\. Table[II](https://arxiv.org/html/2608.11258#S4.T2)shows the final ranking\. The Nemenyi critical difference \(CD\) atα=0\.05\\alpha=0\.05fork=7k=7methods andN=42N=42configurations is 1\.39; no pair of methods exceeds this threshold, indicating that while the overall test rejects the null, individual pairwise differences are not large enough to declare statistical significance by rank alone\.

TABLE II:Final algorithm ranking by Friedman test across all 42 benchmark configurations \(lower rank = better\)\. All adaptive\-learning\-rate hybrids outperform standard PSO\. Adadelta ranks first; SGD ranks last due to divergence on hard functions\.Key takeaway:All adaptive\-LR hybrids beat vanilla PSO\. Adadelta wins because it needs no learning rate, it self\-calibrates\. SGD rankslastbecause it diverges on hard functions\.

![Refer to caption](https://arxiv.org/html/2608.11258v1/x6.png)Figure 6:Performance scores for each method, computed as7−average Friedman rank7\-\\text\{average Friedman rank\}across all 42 benchmark configurations \(higher = better; maximum possible is 6\)\. Five of six AHPSO variants outperform standard PSO\. AHPSO\-SGD scores lowest because its fixed learning rate causes divergence on penalized functions, dragging down its average rank despite strong unimodal performance\.
### IV\-BWhere Hybrids Win

The Friedman test tells us that differences exist, but notwhere\. To pinpoint which functions benefit from hybridization, we run pairwise Mann\-Whitney U tests \(6 methods×\\times42 configurations = 252 tests\)\. Because multiple testing inflates false\-positive rates, atα=0\.05\\alpha=0\.05, we would expect∼13\{\\sim\}13spurious significant results by chance, we apply the Holm\-Bonferroni step\-down correction\[[17](https://arxiv.org/html/2608.11258#bib.bib17)\]\. This procedure sorts all 252pp\-values and rejectsHiH\_\{i\}only ifp\(i\)<α/\(m−i\+1\)p\_\{\(i\)\}<\\alpha/\(m\-i\+1\), controlling the family\-wise error rate while retaining more power than Bonferroni \(Table[III](https://arxiv.org/html/2608.11258#S4.T3)\)\.

TABLE III:Head\-to\-head comparison against standard PSO across 42 configurations\. “Wins” and “Losses” count configurations where the hybrid is significantly better or worse \(Mann\-Whitney U with Holm\-Bonferroni correction atα=0\.05\\alpha=0\.05\)\. Uncorrected counts shown in parentheses\.Of 80 uncorrected significant results, 47 survive Holm\-Bonferroni correction–33 were likely false positives\. The key finding is that AHPSO\-Adadelta has the best net outcome \(\+2\) with only 1 loss after correction, confirming it as the safest hybrid choice\. SGD’s losses \(15\) are overwhelmingly genuine, confirming that fixed learning rates are dangerous in this setting\.

![Refer to caption](https://arxiv.org/html/2608.11258v1/x7.png)Figure 7:Wins and losses by function group\. Unimodal functions show the most wins because their gradients reliably point toward the global optimum\. Multimodal functions are mixed, the sigmoid preserves exploration, but gradient steps occasionally pull particles into wrong basins\. Composite functions \(rotated and shifted combinations\) show mostly ties because rotation destroys axis\-aligned gradient structure, making per\-dimension finite differences less informative\.
### IV\-CHighlight Results

To build intuition forwhyhybrids help or hurt, we examine three representative cases\. For each, we report the rank\-biserial correlationrr​br\_\{rb\}as effect size: it answers “if you pick one run from each method at random, how likely is the hybrid to win?” Values near 1\.0 mean the hybrid always wins; 0\.5 is a large effect; near zero means no practical difference\.

Best case, F1 \(Sphere,dd=30\):On a smooth, bowl\-shaped landscape, gradient descent is devastating\. AHPSO\-SGD finds solutions 27 orders of magnitude better than PSO \(2\.4×10−252\.4\\times 10^\{\-25\}vs\.8\.0×1028\.0\\times 10^\{2\};p=3\.5×10−18p=3\.5\\times 10^\{\-18\},rr​b=1\.00r\_\{rb\}=1\.00\)\. The ideal scenario: the gradient always points toward the optimum, and the adaptive weight lets it dominate once the swarm converges\.

Worst case, F12 \(Penalized,dd=10\):On a function with steep penalty boundaries, SGD’s fixed learning rate causes catastrophic divergence, the gradient step overshoots the penalty wall, lands in an even steeper region, and spirals outward\. AHPSO\-SGD scores7\.6×1077\.6\\times 10^\{7\}vs\. PSO’s6\.2×10−36\.2\\times 10^\{\-3\}\(p=3\.3×10−18p=3\.3\\times 10^\{\-18\}\)\. This is why adaptive optimizers \(Adadelta, Adam\) are safer: they automatically shrink their step size near steep gradients\.

Multimodal success, F11 \(Griewank,dd=30\):This function has many local optima but a clear global structure\. AHPSO\-RMSprop improves on PSO by 79% \(p=9\.5×10−6p=9\.5\\times 10^\{\-6\},rr​b=0\.50r\_\{rb\}=0\.50\)\. The adaptive mechanism works as designed: it keeps gradient influence low while the swarm explores different basins, then ramps it up once particles cluster in the correct region\.

### IV\-DConvergence Behavior and Speed

The previous analysis showsfinalperformance\. But how do algorithms get there? Convergence curves reveal whether hybrids converge faster throughout or only pull ahead late\. We also report a formal convergence speed metric:Evaluations to Target\(ETT\), the total number of function evaluations required to first reach a target accuracy, accounting for the2​d2dgradient evaluations per particle per iteration that AHPSO incurs\.

![Refer to caption](https://arxiv.org/html/2608.11258v1/x8.png)Figure 8:Convergence curves \(median of 15 runs, IQR shading shows 25th–75th percentile\)\. Left: on unimodal F1 \(Sphere,d=10d\{=\}10\), AHPSO\-Adadelta converges faster per iteration but the narrow IQR bands confirm consistent behavior across runs\. Right: on multimodal F9 \(Rastrigin,d=10d\{=\}10\), both methods show wider IQR bands reflecting the stochastic nature of multimodal search, with overlapping confidence regions for the first∼100\{\\sim\}100iterations\.Two patterns emerge \(Fig\.[8](https://arxiv.org/html/2608.11258#S4.F8)\)\. On unimodal F1, the gap between hybrids and PSO growsexponentially, each iteration compounds the advantage because the gradient consistently points toward the optimum\. On multimodal F9, the curves overlap for∼100\{\\sim\}100iterations before separating\. This 100\-iteration overlap is the adaptive mechanism in action: diversity remains high while the swarm explores, keepingα\\alphanear its minimum and preventing premature exploitation\. The separation point corresponds to diversity crossing theτ=0\.3\\tau=0\.3threshold, exactly where the sigmoid transitions from “mostly PSO” to “mostly GD\.”

Convergence speed \(ETT\)\.On F1 \(d=10d\{=\}10, target10−610^\{\-6\}\), PSO reaches the target in a median of 7,890 evaluations versus AHPSO\-Adadelta’s 164,745 evaluations, a 21×\\timesdifference despite AHPSO achieving 12 orders of magnitude betterfinalaccuracy\. Both achieve 100% success rate\. This confirms the budget\-normalized finding: when the target is achievable by PSO alone, the gradient overhead is wasteful\. AHPSO’s value emerges only when PSOcannotreach the target within its budget, or when the final accuracy matters more than time\-to\-threshold\.

### IV\-ESignificance Heatmap

The highlight cases and convergence curves illustratewhyhybrids help or hurt, but only on selected functions\. To verify these patterns hold broadly, we now show the full statistical picture across all 42 configurations at once\.

![Refer to caption](https://arxiv.org/html/2608.11258v1/x9.png)Figure 9:Statistical significance of each hybrid vs\. standard PSO across all 42 configurations \(Mann\-Whitney U,p<0\.05p<0\.05\)\. Green = hybrid significantly better; red = PSO better; gray = no significant difference\. Two patterns are visible: \(1\) unimodal functions \(left columns\) are predominantly green, confirming gradient descent accelerates convergence on smooth landscapes; \(2\) SGD shows a distinctive red band on penalized/composite functions where its fixed step size causes divergence\.Taken together, the results tell a simple story: gradient descent is a powerful but dangerous tool for swarm optimization, and the adaptive sigmoid is the safety mechanism that makes it practical\. Hybridization helps most when the gradient signal is reliable \(unimodal, smooth\), is neutral when exploration dominates \(multimodal\), and hurts only when a fixed\-step optimizer overshoots \(SGD on penalized functions, an optimizer failure, not a mechanism failure\)\.

The practical recommendation follows directly: if you can afford2​d2dextra function evaluations per particle, pair PSO with Adadelta or Adam\. It will either help or be neutral, but rarely hurt\.

### IV\-FCompetitive Baselines

The preceding analysis compares AHPSO variants only against standard PSO, a 30\-year\-old algorithm\. To assess whether the adaptive sigmoid mechanism provides value beyond what existing advanced methods already achieve, we compare against two competitive baselines: CLPSO\[[11](https://arxiv.org/html/2608.11258#bib.bib11)\]and CMA\-ES\[[12](https://arxiv.org/html/2608.11258#bib.bib12)\]\.

CLPSO\(Comprehensive Learning PSO\) prevents premature convergence by having each particle learn from different exemplars per dimension, with tournament\-selected personal bests and a refreshing gap mechanism\.CMA\-ES\(Covariance Matrix Adaptation Evolution Strategy\) adapts a full covariance matrix to capture second\-order landscape information without explicit gradients\. Both use identical evaluation budgets to AHPSO \(30 particles×\\times500 iterations = 15,000 evaluations\)\.

TABLE IV:Friedman average rankings across all 42 configurations with competitive baselines included\. CMA\-ES ranks between AHPSO variants, confirming that AHPSO\-Adadelta’s advantage over vanilla PSO is not trivially achieved by any modern method\.The expanded Friedman test \(χ2​\(8\)=26\.19\\chi^\{2\}\(8\)=26\.19,p=9\.75×10−4p=9\.75\\times 10^\{\-4\}\) confirms significant differences among all nine methods \(Table[IV](https://arxiv.org/html/2608.11258#S4.T4)\)\. AHPSO\-Adadelta retains rank 1, but CMA\-ES \(rank 5, 4\.738\) is highly competitive, positioned between AHPSO\-Adagrad and AHPSO\-RMSprop\. CLPSO \(rank 9, 6\.595\) performs worse than all other methods including vanilla PSO\.

Pairwise Mann\-Whitney comparisons reveal the competitive landscape more precisely:

- •CMA\-ES vs AHPSO\-Adadelta:20 wins, 20 losses, 2 ties, essentially equivalent performance\. CMA\-ES dominates on smooth unimodal functions \(F1–F4, F10–F12\) where its covariance adaptation achieves near\-machine\-precision solutions\. AHPSO\-Adadelta wins on composite functions \(F24–F29\) and noisy landscapes \(F7, F8\) where gradient direction within identified basins outperforms distribution\-based search\.
- •CLPSO vs AHPSO\-Adadelta:7 wins, 29 losses, 6 ties\. CLPSO’s exemplar\-based learning is insufficient to match either gradient\-assisted or covariance\-adapted methods on this benchmark suite\.

Interpretation:AHPSO\-Adadelta and CMA\-ES represent complementary strategies for the same problem, exploiting landscape structure beyond what random perturbation provides\. CMA\-ES uses implicit second\-order information \(covariance\); AHPSO uses explicit first\-order information \(gradients\)\. Their near\-identical overall rankings but different per\-function strengths suggest they exploit different landscape properties\. The practical implication: AHPSO is preferable when gradients are cheap and the landscape has exploitable local structure; CMA\-ES is preferable when the landscape is smooth and unimodal\.

### IV\-GBudget\-Normalized Comparison

A critical concern with the preceding analysis isevaluation budget fairness\. AHPSO uses2​d2dadditional function evaluations per particle per iteration for numerical gradient computation\. Atd=30d=30with 30 particles and 500 iterations, this amounts to 915,000 total evaluations versus PSO’s 15,000, a 61×\\timesdisparity\. The iteration\-matched comparison above may therefore conflate the benefit of gradient information with the benefit of simply evaluating the function more often\.

To isolate the contribution of gradientdirectionfrom gradientcost, we give vanilla PSO an equivalent evaluation budget: 30,500 iterations atd=30d=30\(915,000 evaluations\) and 10,500 iterations atd=10d=10\(315,000 evaluations\)\. All other parameters remain identical \(N=30N=30,w∈\[0\.4,0\.9\]w\\in\[0\.4,0\.9\],c1=c2=2\.0c\_\{1\}=c\_\{2\}=2\.0\)\.

TABLE V:Budget\-normalized comparison across 40 configurations\. PSOBN\{\}\_\{\\text\{BN\}\}= PSO with equivalent total function evaluations as AHPSO\. When given the same computational budget, PSO dominates on most configurations\. AHPSO retains advantage only where gradient direction provides information beyond what additional random sampling achieves\.Result:Under equal budgets, PSOBN\{\}\_\{\\text\{BN\}\}wins 21/40 configurations, AHPSO wins 8/40, and 11 are ties \(Table[V](https://arxiv.org/html/2608.11258#S4.T5)\)\. The Friedman test now strongly favors PSOBN\{\}\_\{\\text\{BN\}\}\(χ2​\(6\)=28\.69\\chi^\{2\}\(6\)=28\.69,p=7\.0×10−5p=7\.0\\times 10^\{\-5\}\), with PSOBN\{\}\_\{\\text\{BN\}\}achieving average rank 2\.60 versus AHPSO\-Adadelta’s 3\.88\.

Interpretation:On smooth unimodal functions \(F1–F4\), extra iterations alone suffice, PSO converges to machine precision given enough time, making gradient information redundant\. On multimodal functions with exploitable local structure \(F8, F24–F27\), gradient direction provides genuine value that random sampling cannot replicate\. These are functions where the landscape has smooth basins that reward precise local descent once the correct basin is found\.

Revised claim:The adaptive sigmoid mechanism does not provide a universal improvement over PSO when evaluation budgets are equalized\. Its value isconditional: on problems where \(1\) the landscape has smooth local structure exploitable by gradients, and \(2\) the correct basin has been identified by PSO’s exploration phase, the directed gradient step converges faster than undirected random sampling\. The mechanism’s contribution is therefore best characterized asconverting function evaluations into directed information, valuable when gradients are informative, wasteful when they are not\.

What remains is to assess parameter sensitivity and the broader implications\.

### IV\-HSensitivity Analysis

The sigmoid mechanism introduces three parameters: thresholdτ\\tau, steepnesskk, and minimum weightαmin\\alpha\_\{\\min\}\. We fixαmin=0\.1\\alpha\_\{\\min\}=0\.1\(ensuring gradients never fully dominate\) and sweepτ∈\{0\.1,0\.2,0\.3,0\.4,0\.5\}\\tau\\in\\\{0\.1,0\.2,0\.3,0\.4,0\.5\\\}andk∈\{3,5,10\}k\\in\\\{3,5,10\\\}on four representative functions \(F1, F9, F10, F11\) atd=10d=10with 15 runs each, using AHPSO\-Adadelta\.

TABLE VI:Average rank across 4 functions for differentτ\\tauvalues \(k=5k=5fixed\)\. Lower is better\. Rankings are stable acrossτ∈\[0\.2,0\.4\]\\tau\\in\[0\.2,0\.4\], confirming the mechanism is not sensitive to precise threshold placement\.TABLE VII:Average rank across 4 functions for differentkkvalues \(τ=0\.3\\tau=0\.3fixed\)\. Softer transitions \(k=3k=3\) slightly outperform sharper ones \(k=10k=10\), but differences are small\.Result:Rankings are stable acrossτ∈\[0\.2,0\.4\]\\tau\\in\[0\.2,0\.4\]\(Table[VI](https://arxiv.org/html/2608.11258#S4.T6)\), with extreme values \(τ=0\.1\\tau=0\.1: gradients too early;τ=0\.5\\tau=0\.5: gradients too late\) performing worst\. For steepness \(Table[VII](https://arxiv.org/html/2608.11258#S4.T7)\), softer transitions \(k=3k=3\) marginally outperform sharper ones \(k=10k=10\), suggesting that a gradual blend is preferable to a hard switch\. The defaultτ=0\.3\\tau=0\.3,k=5k=5is near\-optimal but not uniquely so, practitioners can safely use anyτ∈\[0\.2,0\.4\]\\tau\\in\[0\.2,0\.4\]without retuning\.

### IV\-IReal\-World Engineering Application

To validate practical relevance beyond synthetic benchmarks, we apply all methods to two classical constrained engineering design problems from Coello\[[19](https://arxiv.org/html/2608.11258#bib.bib19)\]\.

#### IV\-I1Problem Formulations

Welded Beam Design\.Minimize fabrication cost of a welded beam with 4 variables \(weld thicknesshh, weld lengthll, beam heighttt, beam widthbb\) subject to shear stress \(τ≤13,600\\tau\\leq 13\{,\}600psi\), bending stress \(σ≤30,000\\sigma\\leq 30\{,\}000psi\), buckling load \(Pc≥6,000P\_\{c\}\\geq 6\{,\}000lb\), and deflection \(δ≤0\.25\\delta\\leq 0\.25in\) constraints\. Known optimum:f∗≈1\.7248f^\{\*\}\\approx 1\.7248\.

Pressure Vessel Design\.Minimize total cost \(material \+ forming \+ welding\) of a cylindrical vessel with 4 variables \(shell thicknessTsT\_\{s\}, head thicknessThT\_\{h\}, inner radiusRR, lengthLL\) subject to 4 constraints on minimum thickness and volume\. Known optimum:f∗≈5868\.76f^\{\*\}\\approx 5868\.76\.

Both use quadratic penalty \(λ=106\\lambda=10^\{6\}\) for constraint violations\. Settings: 30 particles, 500 iterations, 50 independent runs\.

#### IV\-I2Results

TABLE VIII:Engineering design optimization results \(50 runs\)\. Best known: Welded Beam≈1\.7248\\approx 1\.7248, Pressure Vessel≈5868\.76\\approx 5868\.76\.Table[VIII](https://arxiv.org/html/2608.11258#S4.T8)shows results\. CMA\-ES dominates both problems, converging reliably to near\-optimal solutions with negligible variance, consistent with its known strength on low\-dimensional \(d≤10d\\leq 10\) smooth problems\. AHPSO\-Adadelta outperforms vanilla PSO on Welded Beam \(lower mean and variance\), confirming that gradient information helps navigate the smooth feasible region\. On Pressure Vessel, CLPSO’s exemplar\-based learning proves more effective than gradient injection for handling the narrow feasible corridor\.

These results reinforce our benchmark findings: AHPSO’s gradient mechanism provides value on problems with smooth local structure, but the advantage is modest on low\-dimensional problems where CMA\-ES’s covariance adaptation is more powerful\. The practical niche for AHPSO lies in medium\-dimensional \(10≤d≤3010\\leq d\\leq 30\) problems where CMA\-ES’sO​\(d2\)O\(d^\{2\}\)covariance update becomes expensive but gradient direction still provides useful exploitation signal\.

### IV\-JAblation Study: Does the Sigmoid Add Value?

The adaptive sigmoid \(Eq\.[5](https://arxiv.org/html/2608.11258#S2.E5)\) is the core novelty, but does it actually outperform a fixed gradient weight? To answer this, we compare AHPSO\-Adadelta with the adaptive sigmoid against five fixed\-α\\alphavariants \(α∈\{0\.1,0\.3,0\.5,0\.7,1\.0\}\\alpha\\in\\\{0\.1,0\.3,0\.5,0\.7,1\.0\\\}\) on four representative functions \(F1, F9, F11, F12 atd=10d\{=\}10, 50 runs each\)\.

TABLE IX:Ablation: adaptive sigmoid vs\. fixedα\\alpha\(AHPSO\-Adadelta,d=10d\{=\}10, 50 runs\)\. Median fitness reported\. Bold = best per function\. The adaptive sigmoid achieves best or near\-best performance across all function types without requiringα\\alphaselection\.Table[IX](https://arxiv.org/html/2608.11258#S4.T9)reveals a nuanced picture\. On unimodal F1, the adaptive sigmoid achieves the best median by a wide margin \(2\.6e\-16 vs\. 5\.9e\-15 forα=1\.0\\alpha\{=\}1\.0\), the early low\-α\\alphaphase preserves exploration breadth before committing to gradient refinement\. On penalized F12, the sigmoid also wins, confirming that ramping gradient influence gradually avoids the divergence that high fixedα\\alphacan cause on steep boundaries\.

On multimodal F9 and F11, fixedα\\alphavalues \(0\.3 and 0\.5 respectively\) slightly outperform the sigmoid\. This is expected: on these functions, the optimal gradient weight is problem\-specific, and a fixed value tuned to that problem will beat a general\-purpose adaptive mechanism\. The sigmoid’s value is not per\-function optimality butrobustness across function types, it achieves competitive performance on all four without any tuning, whereas each fixedα\\alphaexcels on one type but underperforms on others\.

### IV\-KWall\-Clock Time

The evaluation budget analysis \(Section[IV\-G](https://arxiv.org/html/2608.11258#S4.SS7)\) quantifies computational cost in function evaluations\. Table[X](https://arxiv.org/html/2608.11258#S4.T10)translates this to wall\-clock time on commodity hardware \(single\-threaded Python, Intel Xeon, no GPU\)\.

TABLE X:Wall\-clock time per single optimization run \(F1 Sphere, 30 particles, 500 iterations\)\. AHPSO’s gradient computation dominates runtime at higher dimensions due to2​d2dfinite\-difference evaluations per particle\.AHPSO is∼20×\{\\sim\}20\\timesslower than PSO atd=10d\{=\}10and∼50×\{\\sim\}50\\timesslower atd=30d\{=\}30, consistent with the2​d2dgradient overhead\. For cheap objective functions \(milliseconds per evaluation\), this overhead is negligible in absolute terms \(<10<10s per run\)\. For expensive simulations, the overhead becomes prohibitive, reinforcing that AHPSO’s practical niche is problems where function evaluations are cheap but gradient direction is informative\.

### IV\-LCLPSO Performance Note

CLPSO ranks last \(9th\) in our expanded comparison \(Table[IV](https://arxiv.org/html/2608.11258#S4.T4)\), below vanilla PSO\. This counterintuitive result deserves explanation\. CLPSO’s exemplar\-based learning excels at maintaining diversity on multimodal functions \(it wins on F8 and F9\), but its per\-particle\-per\-dimension exemplar selection creates overhead that slows convergence on unimodal functions where standard PSO’s social learning is sufficient\. Additionally, CLPSO’s refreshing gap mechanism \(m=7m\{=\}7stagnation iterations before re\-selecting exemplars\) can delay adaptation on penalized functions \(F12, F13\) where rapid response to boundary violations is critical\. The result is consistent with Liang et al\.’s original findings: CLPSO was designed for multimodal optimization specifically, not as a general\-purpose improvement over PSO\.

## VConclusion

We asked whether a simple diversity\-based switch could combine PSO’s exploration with gradient descent’s precision, without manual tuning of when to transition\. Across 29 benchmark functions, 2 engineering design problems, 42 configurations, and 14,700\+ independent runs, compared against both vanilla PSO and competitive baselines \(CLPSO, CMA\-ES\), the answer isconditionallyyes, with five main findings:

1. 1\.Under iteration\-matched comparison, Adadelta is the best partner for PSO\.It needs no learning rate and self\-adapts to any landscape\. Rank 1 of 9 methods \(p=9\.75×10−4p=9\.75\\times 10^\{\-4\}, Friedman\)\.
2. 2\.AHPSO\-Adadelta is competitive with CMA\-ES\.Against the gold\-standard continuous optimizer, AHPSO\-Adadelta achieves 20 wins, 20 losses, and 2 ties, near\-equivalent overall performance with complementary per\-function strengths\.
3. 3\.Under budget\-normalized comparison, PSO dominates\.When given equivalent total function evaluations \(61×\\timesmore iterations\), PSO wins 52\.5% of configurations versus AHPSO’s 20% \(p=7\.0×10−5p=7\.0\\times 10^\{\-5\}, Friedman\)\.
4. 4\.Gradient direction has conditional value\.AHPSO retains advantage on problems with smooth local basins \(F8, F24–F27\) where directed descent outperforms undirected sampling, even at equal cost\.
5. 5\.The sigmoid mechanism correctly gates gradient influence\.It preserves exploration on multimodal functions and enables exploitation on unimodal ones, but this benefit is insufficient to overcome the 61×\\timesevaluation overhead on most problems\.

Limitations\.

1. 1\.Computational overhead\.Estimating gradients via central differences \(Eq\.[7](https://arxiv.org/html/2608.11258#S2.E7)\) costs2​d2dextra function evaluations per particle per iteration\. For our largest setting \(d=30d\{=\}30, 30 particles\), that is 1,800 extra evaluations per iteration on top of the 30 baseline evaluations PSO already performs, a61×61\\timesincrease in function calls\. This is acceptable whenffis cheap \(milliseconds per call, as in our benchmarks\), but prohibitive for expensive simulations where each evaluation takes minutes or hours\.
2. 2\.Empirical threshold\.The sigmoid midpointτ=0\.3\\tau\{=\}0\.3was tuned on this benchmark suite and may need adjustment for other problem classes\.
3. 3\.Differentiability assumption\.Finite\-difference gradients require continuous functions\. Discontinuous or combinatorial landscapes would produce misleading gradient signals, making the method inapplicable without modification\.
4. 4\.Dual learning rate\.Our experiments useη=0\.01\\eta=0\.01for unimodal andη=0\.001\\eta=0\.001for multimodal functions, requiring problem\-class knowledge a priori\. This constitutes oracle information unavailable in real applications\. Adadelta sidesteps this entirely \(it ignoresη\\eta\), but SGD\-based variants are sensitive to this choice\. A singleη=0\.01\\eta=0\.01for all functions would degrade multimodal performance; we chose to report the best\-case for each optimizer to characterize upper bounds\.

Future work\.

1. 1\.Let each particle choose its own optimizer\.Our results show that different optimizers excel on different landscape types\. A natural extension is to let each particle select its optimizer adaptively, particles in smooth regions would use SGD \(fast\), while particles near steep boundaries would switch to Adadelta \(safe\)\. Multi\-armed bandit algorithms could automate this selection based on each particle’s recent improvement history\.
2. 2\.Higher\-dimensional engineering problems\.Our engineering validation used 4\-variable problems where CMA\-ES dominates\. Problems withd≥20d\\geq 20design variables \(e\.g\., topology optimization, neural architecture search\) would better showcase AHPSO’s gradient advantage\.
3. 3\.Eliminate the gradient cost\.When the objective function is available as source code \(not a black\-box simulator\), automatic differentiation can compute exact gradients inO​\(1\)O\(1\)passes rather than2​d2dfinite\-difference evaluations\. This would make AHPSO practical even for high\-dimensional problems \(d\>100d\>100\)\.

## References

- \[1\]J\. Kennedy and R\. Eberhart, “Particle swarm optimization,” inProc\. IEEE ICNN, 1995, pp\. 1942–1948\.
- \[2\]S\. Mirjalili, S\. M\. Mirjalili, and A\. Lewis, “Grey wolf optimizer,”Advances in Engineering Software, vol\. 69, pp\. 46–61, 2014\.
- \[3\]D\. P\. Kingma and J\. Ba, “Adam: A method for stochastic optimization,” inProc\. ICLR, 2015\.
- \[4\]M\. D\. Zeiler, “ADADELTA: An adaptive learning rate method,”arXiv:1212\.5701, 2012\.
- \[5\]J\. Duchi, E\. Hazan, and Y\. Singer, “Adaptive subgradient methods for online learning and stochastic optimization,”JMLR, vol\. 12, pp\. 2121–2159, 2011\.
- \[6\]T\. Tieleman and G\. Hinton, “RMSProp: Divide the gradient by a running average of its recent magnitude,”COURSERA: Neural Networks for Machine Learning, Lecture 6\.5, 2012\.
- \[7\]T\. Dozat, “Incorporating Nesterov momentum into Adam,” inICLR Workshop, 2016\.
- \[8\]M\. M\. Noel and T\. C\. Jannett, “Simulation of a new hybrid particle swarm optimization algorithm,” inProc\. 36th Southeastern Symp\. System Theory, 2004, pp\. 150–153\.
- \[9\]Y\. Shi and R\. Eberhart, “A modified particle swarm optimizer,” inProc\. IEEE Int\. Conf\. Evol\. Comput\., 1998, pp\. 69–73\.
- \[10\]Z\.\-H\. Zhan, J\. Zhang, Y\. Li, and H\. S\.\-H\. Chung, “Adaptive particle swarm optimization,”IEEE Trans\. Syst\., Man, Cybern\. B, vol\. 39, no\. 6, pp\. 1362–1381, 2009\.
- \[11\]J\. J\. Liang, A\. K\. Qin, P\. N\. Suganthan, and S\. Baskar, “Comprehensive learning particle swarm optimizer for global optimization of multimodal functions,”IEEE Trans\. Evol\. Comput\., vol\. 10, no\. 3, pp\. 281–295, 2006\.
- \[12\]N\. Hansen, “The CMA evolution strategy: A comparing review,” inTowards a New Evolutionary Computation, J\. A\. Lozano et al\., Eds\. Berlin: Springer, 2006, pp\. 75–102\.
- \[13\]R\. Tanabe and A\. S\. Fukunaga, “Improving the search performance of SHADE using linear population size reduction,” inProc\. IEEE CEC, 2014, pp\. 1658–1665\.
- \[14\]Y\. S\. Ong and A\. J\. Keane, “Meta\-Lamarckian learning in memetic algorithms,”IEEE Trans\. Evol\. Comput\., vol\. 8, no\. 2, pp\. 99–110, 2004\.
- \[15\]A\. Ratnaweera, S\. K\. Halgamuge, and H\. C\. Watson, “Self\-organizing hierarchical particle swarm optimizer with time\-varying acceleration coefficients,”IEEE Trans\. Evol\. Comput\., vol\. 8, no\. 3, pp\. 240–255, 2004\.
- \[16\]S\. K\. S\. Fan and E\. Yan, “A hybrid particle swarm optimization with local search strategy,”Applied Soft Computing, vol\. 27, pp\. 459–472, 2015\.
- \[17\]S\. Holm, “A simple sequentially rejective multiple test procedure,”Scandinavian Journal of Statistics, vol\. 6, no\. 2, pp\. 65–70, 1979\.
- \[18\]M\. Friedman, “The use of ranks to avoid the assumption of normality implicit in the analysis of variance,”J\. Amer\. Statist\. Assoc\., vol\. 32, no\. 200, pp\. 675–701, 1937\.
- \[19\]C\. A\. Coello Coello, “Use of a self\-adaptive penalty approach for engineering optimization problems,”Computers in Industry, vol\. 41, no\. 2, pp\. 113–127, 2000\.
- \[20\]J\. Derrac, S\. García, D\. Molina, and F\. Herrera, “A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms,”Swarm and Evolutionary Computation, vol\. 1, no\. 1, pp\. 3–18, 2011\.
- \[21\]M\. R\. Bonyadi and Z\. Michalewicz, “Particle swarm optimization for single objective continuous space problems: A review,”Evolutionary Computation, vol\. 25, no\. 1, pp\. 1–54, 2017\.
- \[22\]M\. G\. Epitropakis, V\. P\. Plagianakos, and M\. N\. Vrahatis, “Evolving cognitive and social experience in particle swarm optimization through differential evolution: A hybrid approach,”Information Sciences, vol\. 216, pp\. 50–92, 2012\.
- \[23\]P\. A\. N\. Bosman and E\. D\. de Jong, “Combining gradient techniques for numerical multi\-objective evolutionary optimization,” inProc\. GECCO, 2005, pp\. 627–634\.
- \[24\]W\. H\. Lim and N\. A\. M\. Isa, “Two\-layer particle swarm optimization with intelligent division of labor,”Engineering Applications of Artificial Intelligence, vol\. 26, no\. 10, pp\. 2263–2279, 2014\.
- \[25\]D\. H\. Wolpert and W\. G\. Macready, “No free lunch theorems for optimization,”IEEE Trans\. Evol\. Comput\., vol\. 1, no\. 1, pp\. 67–82, 1997\.