IBM WebSphere environments are notorious for generating dynamic URLs that confuse search engine bots. This section focuses on stabilizing your crawl environment.
Audit and Strip jsessionid from Public URLs WebSphere often appends session IDs to URLs. If these are indexed, they create infinite duplicate content loops. Configure the WAS deployment descriptor (web.xml) to use cookies for session tracking instead of URL rewriting. Tools: Screaming Frog, Google Search Console, IBM Integrated Solutions Console
Configure DynaCache for Search Bot User-Agents Ensure that the IBM Dynamic Cache (DynaCache) is not serving stale content to Googlebot while serving fresh content to users. This can lead to 'cloaking' flags in search audits. Tools: IBM Cache Monitor, Custom Bot Headers
Implement XML Sitemap Generation for WAS Dynamic Content Enterprise systems often have millions of pages. Use a custom Java-based sitemap generator that hooks into the WAS data layer to provide real-time updates to search engines. Tools: Java Sitemap API, Google Search Console
Checklist item: Remove jsessionid from public crawl paths. Evidence required: crawl exports showing sampled public URLs, response headers, redirects, canonical elements, and the WebSphere session-tracking configuration that applies to those requests.
Pass or fail condition: pass when intended public content has a stable cookie-capable URL path and sampled navigation does not expose session identifiers in crawlable links or redirects; fail when session rewriting creates alternate public URLs or conflicting index signals.
Severity: high when the pattern scales across templates or navigation. Owner: WebSphere application and platform owners. Corrective action: adjust session tracking and application URL generation so public navigation does not depend on rewritten session identifiers.
Validation step: repeat the crawl from a clean session, inspect server responses, and confirm the canonical target and internal links remain stable.
Checklist item: Verify DynaCache response consistency. Evidence required: paired request samples, cache status evidence, timestamps or freshness indicators available to the team, and the resulting HTML for the same public URL under comparable conditions.
Pass or fail condition: pass when cache behavior serves the intended public representation and does not vary indexable content merely because a request identifies as a crawler; fail when stale, incomplete, or session-dependent variants alter material indexable content.
Severity: high for shared templates, medium for isolated pages. Owner: WebSphere platform and application teams. Corrective action: revise cache keys, invalidation, personalization boundaries, or bypass rules based on the observed defect.
Validation step: clear or age the relevant cache as appropriate, repeat the paired requests, and compare status, headers, and HTML.
Checklist item: Validate XML sitemap coverage for dynamic public content. Evidence required: the generated sitemap output, a current inventory of intended indexable URLs, status checks for submitted URLs, and Search Console reporting where available.
Pass or fail condition: pass when sitemap entries represent canonical, indexable public URLs and removed or redirected URLs are not continually republished; fail when the sitemap contains sessionized, duplicate, blocked, redirected, or non-public destinations.
Severity: medium, or high when discovery depends heavily on generated inventories. Owner: application engineering with search operations. Corrective action: change the sitemap generation logic so it reads the same publish state and canonical URL rules used by the public application.
Validation step: regenerate the sitemap, sample entries against live responses and index directives, then resubmit through Google Search Console.