bug: e2e tests failing when updating to 4.20.0

[danyball]

Prerequisites

• I have read the Contributing Guidelines.
• I agree to follow the Code of Conduct.
• I have searched for existing issues that already report this problem, without success.

Stencil Version

4.20.0

Current Behavior

We have around 700 e2e tests running with 4.19.2. After updating to 4.20.0 tests fails on our server (jenkins and gitlab ci tested). On my local Mac I can not see any errors.

• some of them "App did not load in allowed time. Please ensure the content loads a stencil application."
• some with random errors.
• the test script just stopped at a point - no more logging / gitlab stops those jobs after 1h

Expected Behavior

tests still running

System Info

Server:
- tested with several node versions, from 18 to LTS
- debian bookworm OS
- HeadlessChrome/121.0.6167.85 (with puppeteer 21.11.0 - also tested 21.0.0)
- on gitlab ci:
   - `- apt-get update && apt-get install -y libnss3 libdbus-1-3 libatk1.0-0 libx11-xcb1 libxcomposite1 libxcursor1 libxdamage1 libxi6 libxtst6 libxrandr2 libasound2 libatk-bridge2.0-0 libgtk-3-0 libcups2 libpango1.0-0 libxss1 fonts-liberation xdg-utils
`

local:
System: node 18.20.5 (LTS also working)
    Platform: darwin (24.1.0)
   CPU Model: Apple M1 Max (10 cpus)
    Compiler: /node_modules/@stencil/core/compiler/stencil.js
       Build: 1732216626
     Stencil: 4.22.3 🐤
  TypeScript: 5.5.4
      Rollup: 2.56.3
      Parse5: 7.1.2
      jQuery: 4.0.0-pre
      Terser: 5.31.1

Steps to Reproduce

• having many tests (do not find a reproducer)
• use specific stencil version
• run e2e tests on a server env

Code Reproduction URL

sorry no

Additional Information

https://discord.com/channels/520266681499779082/1304108899657318442/1304108899657318442

[christian-bromann]

Hey @danyball 👋 when loading a Stencil component using the Stencil testrunner in an e2e test case we verify that all components resolved their componentOnReady hook. If this has not happened before the config.testing.browserWaitUntil timeout is reached it fails with given error.

At this point there is not much I can do as this is very difficult to reproduce. If you have any ideas what may can cause this I would be curious to know.

[danyball]

Thanks Christian. Did you change something runtime timing relevant in 4.20?
Maybe something in

runtime: hydrate shadow dom first (#5911) (ccf1a89)
runtime: make isSameVnode return false on initial render in a hydration case (#5891) (82a7bb9)
?

Maybe the new if in "isSameVnode" could go wrong on slow machines or so?!

[christian-bromann]

@danyball potentially but I am not certain.

[tishoyanchev]

We are facing the same or a similar issue. @christian-bromann, @danyball
Two days ago the e2e unit tests started failing the checks on Github actions. When I run them locally, all of them fail with the error "App did not load in allowed time. Please ensure the content loads a stencil application."
On Github Actions, the log is showing this error:

Puppeteer old Headless deprecation warning:
    In the near future `headless: true` will default to the new Headless mode
    for Chrome instead of the old Headless implementation. For more
    information, please see https://developer.chrome.com/articles/new-headless/.
    Consider opting in early by passing `headless: "new"` to `puppeteer.launch()`
    If you encounter any bugs, please report them to https://github.com/puppeteer/puppeteer/issues/new/choose.
[16:09.3]  transpile started ...
[ ERROR ]  Failed to launch the browser process!
           [0115/151609.612447:FATAL:zygote_host_impl_linux.cc(1[27](https://github.com/Infineon/infineon-design-system-stencil/actions/runs/12791252232/job/35658932635#step:5:28))] No usable
           sandbox! Update your kernel or see
           https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md
           for more information on developing with the SUID sandbox. If you
           want to live dangerously and need an immediate workaround, you can
           try using --no-sandbox. TROUBLESHOOTING:
           https://pptr.dev/troubleshooting Error: Failed to launch the browser
           process! [0115/151609.612447:FATAL:zygote_host_impl_linux.cc(127)]
           No usable sandbox! Update your kernel or see
           https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md
           for more information on developing with the SUID sandbox.

My stencil version is 4.23.0
My puppeteer version is 21.9.0

Any ideas what to do?

[christian-bromann]

@tishoyanchev we are seeing the same issues in our pipeline and investigating the issue.

[tishoyanchev]

@tishoyanchev we are seeing the same issues in our pipeline and investigating the issue.

@christian-bromann
Hi, I saw that you guys released a new version for Stencil that is meant to fix this problem, however, I still have it.
Here's a more detailed explanation of the problem I'm facing:

• Some months ago our dependabot created a PR for puppeteer's new version. We tested it, and found out that it's not compatible with Stencil, and the test runners were failing with the same error described in this Issue. So we decided to keep using their old version ^21.9.0 until they fix their problem.
• Few days ago, the e2e unit tests check on GitHub actions started failing. When I tested it locally, all components failed with the above described error.
• Yesterday I tried removing the ^ symbol on puppeteer because I suspected a newer version could have been applied. I just installed the fixed 21.9.0 version, and surprisingly all tests passed locally! but, they still failed on github actions, although this time, only 9 were failing, while the rest were passing. I cleaned cache, but it didn't make any difference.
• Today I updated stencil to your latest version, and tried with puppeteer version ^21.9.0, 21.9.0 and latest 24.1.0, but 9 tests keep failing locally and on github actions. The error is the same: App did not load within 10000ms. Please ensure the content loads a stencil application.

Any ideas what to do?
Is it working for you? With what version?

[thatdom]

we have same issue

[christian-bromann]

@tishoyanchev can anyone provide a reproduction case?

[tishoyanchev]

@tishoyanchev can anyone provide a reproduction case?

@christian-bromann

"@stencil/core": "^4.24.0",
"puppeteer": "^24.1.0",

Example of a test that fails:

Component.tsx:
//ifxTabs.tsx

import { Component, h, Prop, State, Element, Listen, Event, EventEmitter, Watch } from '@stencil/core';
@Component({
  tag: 'ifx-tabs',
  styleUrl: 'tabs.scss',
  shadow: true
})
export class IfxTabs {
  @Element() el: HTMLElement;

  @Prop() orientation: string = "horizontal";
  @Prop({ mutable: true }) activeTabIndex: number = 0;
  @Prop() fullWidth: boolean = false;

  @State() internalOrientation: string;
  @State() internalActiveTabIndex: number = 0;
  @State() internalFocusedTabIndex: number = 0;
  @State() tabRefs: HTMLElement[] = [];
  @State() tabHeaderRefs: HTMLElement[] = [];
  @State() disabledTabs: string[] = [];
  @State() tabObjects: any[] = [];

  @Event() ifxTabChange: EventEmitter;

  @Listen('resize', {target: 'window'})
  updateBorderOnWindowResize() {
    this.updateBorderAndFocus();
  }

  setActiveAndFocusedTab(index: number) {
    if (index >= this.tabObjects.length) {
      index = this.tabObjects.length - 1;
    }
    if (index < 0) {
      index = 0;
    }
    if (!this.tabObjects[index]?.disabled) {
      this.internalActiveTabIndex = index;
      this.internalFocusedTabIndex = index;
    }
  }

  @Listen('tabHeaderChange')
  handleTabHeaderChange(e) { 
    const tabIndex = e.target.getAttribute('slot').replace('tab-', '');
    this.tabObjects[tabIndex].header = e.detail;
    this.tabObjects = [...this.tabObjects]; 
  }
  

  @Watch('activeTabIndex')
  activeTabIndexChanged(newValue: number, oldValue: number) {
    if (newValue !== oldValue) {
      this.setActiveAndFocusedTab(newValue);
    }
  }



  componentWillLoad() {
    this.internalOrientation = this.orientation.toLowerCase() === 'vertical' ? 'vertical' : 'horizontal';
    if (this.internalActiveTabIndex !== this.activeTabIndex) {
      this.ifxTabChange.emit({ previousTab: this.internalActiveTabIndex, currentTab: this.activeTabIndex });
    };
    this.onSlotChange();
    this.setActiveAndFocusedTab(this.activeTabIndex);
    this.updateTabStyles();
  }

  updateTabStyles() {
    this.tabHeaderRefs.forEach((tab, index) => {
      tab.classList.toggle('active', index === this.internalActiveTabIndex);
      tab.setAttribute('aria-selected', index === this.internalActiveTabIndex ? 'true' : 'false')
    });
  }


  // needed for smooth border transition
  reRenderBorder() {
    const borderElement = this.el.shadowRoot.querySelector('.active-border') as HTMLElement;
    if (borderElement && this.tabHeaderRefs[this.internalActiveTabIndex]) {
      if (this.orientation === 'horizontal') {

        borderElement.style.left = `${this.tabHeaderRefs[this.internalActiveTabIndex].offsetLeft}px`;
        borderElement.style.width = `${this.tabHeaderRefs[this.internalActiveTabIndex].offsetWidth}px`;
        borderElement.style.top = '';
        borderElement.style.height = '';
      } else {
        borderElement.style.top = `${this.tabHeaderRefs[this.internalActiveTabIndex].offsetTop}px`;
        borderElement.style.height = `${this.tabHeaderRefs[this.internalActiveTabIndex].offsetHeight}px`;
        borderElement.style.left = '';
        borderElement.style.width = '';
      }
    }
  }


  // when a slot is removed / added
  @Listen('slotchange')
  onSlotChange() {
    const tabs = this.el.querySelectorAll('ifx-tab');
    this.tabObjects = Array.from(tabs).map((tab) => {
      return {
        header: tab?.header,
        disabled: tab?.disabled === true,
        icon: tab?.icon,
        iconPosition: tab?.iconPosition
      }
    });

    this.tabRefs = Array.from(tabs);
    this.tabRefs.forEach((tab, index) => {
      tab.setAttribute('slot', `tab-${index}`);
    });
  }

  setDefaultOrientation() {
    const validOrientations = ['horizontal', 'vertical'];
    const lowercaseOrientation = this.orientation.toLowerCase();

    if (!validOrientations.includes(lowercaseOrientation)) {
      this.internalOrientation = 'horizontal';
    } else this.internalOrientation = this.orientation;
  }

  componentDidLoad() {
    this.updateBorderAndFocus();
    // Add keyboard event listeners for each tab header
    this.tabHeaderRefs.forEach((tab, index) => {
      tab.addEventListener('focus', this.onTabFocus(index));
    });

  }

  onTabFocus(index) {
    return () => {
      this.internalFocusedTabIndex = index;
    };
  }

  disconnectedCallback() {
    // Remove keyboard event listeners when component is unmounted
    this.tabHeaderRefs.forEach((tab, index) => {
      tab.removeEventListener('focus', this.onTabFocus(index));
    });
  }
  componentDidUpdate() {
    this.updateBorderAndFocus();
  }

  private updateBorderAndFocus() {
    this.reRenderBorder()
    this.updateTabFocusability();
  }

  private updateTabFocusability() {
    this.tabHeaderRefs.forEach((tab, index) => {
      tab.tabIndex = index === this.internalActiveTabIndex ? 0 : -1;
    })
  }


  private focusNextTab() {
    let nextIndex = this.internalFocusedTabIndex + 1;
    while (nextIndex < this.tabHeaderRefs.length && this.tabObjects[nextIndex].disabled) {
      nextIndex++;
    }
    if (nextIndex >= 0 && nextIndex < this.tabHeaderRefs.length) {
      this.internalFocusedTabIndex = nextIndex;
      this.tabHeaderRefs[nextIndex].focus();
    }
  }

  private focusPreviousTab() {
    let prevIndex = this.internalFocusedTabIndex - 1;
    while ((prevIndex >= 0) && (this.tabObjects[prevIndex].disabled)) {
      prevIndex--;
    }
    if ((prevIndex >= 0) && (prevIndex < this.tabHeaderRefs.length)) {
      this.internalFocusedTabIndex = prevIndex;
      this.tabHeaderRefs[prevIndex].focus();
    }
  }


  private getTabItemClass(index: number) {
    const isActive = index === this.internalActiveTabIndex && !this.tabObjects[index].disabled;
    const isDisabled = this.tabObjects[index].disabled;
    const iconPosition = this.tabObjects[index].iconPosition
    return `tab-item ${this.fullWidth ? 'full-width' : ""} ${isActive ? 'active' : ''} ${isDisabled ? 'disabled' : ''} ${'icon__'+iconPosition}`;
  }

  private handleClick(tab, index) {
    this.ifxTabChange.emit({ previousTab: this.internalActiveTabIndex, currentTab: index })
    if (!tab.disabled) this.internalActiveTabIndex = index;

  }



  @Listen('keydown')
  handleKeyDown(ev: KeyboardEvent) {
    if (ev.key === 'Tab') {

      if (ev.shiftKey) {
        // Shift + Tab
        if (this.internalFocusedTabIndex === 0) {
          // Allow default behavior to move focus out of component
          return;
        } else {
          ev.preventDefault();
          this.focusPreviousTab();
        }
      } else {
        // Tab
        if (this.internalFocusedTabIndex === this.tabHeaderRefs.length - 1) {
          // Allow default behavior to move focus out of component
          return;
        } else {
          ev.preventDefault();
          this.focusNextTab();
        }
      }
    } else if (ev.key === 'Enter') {
      if (this.internalFocusedTabIndex !== -1 && !this.tabObjects[this.internalFocusedTabIndex].disabled) {
        const previouslyActiveTabIndex = this.internalActiveTabIndex;
        this.internalActiveTabIndex = this.internalFocusedTabIndex;
        this.ifxTabChange.emit({ previousTab: previouslyActiveTabIndex, currentTab: this.internalFocusedTabIndex })
      }
    }
  }


  render() {
    return (
      <div aria-label="navigation tabs" class={`tabs ${this.internalOrientation}`}>
        <ul role="tablist" class="tabs-list">
          {this.tabObjects?.map((tab, index) => (
            <li
              class={this.getTabItemClass(index)}
              ref={(el) => (this.tabHeaderRefs[index] = el)}
              onMouseDown={(event) => event.preventDefault()}
              onClick={() => this.handleClick(tab, index)}
              aria-selected={index === this.internalActiveTabIndex ? 'true' : 'false'}
              aria-disabled={tab.disabled ? 'true' : 'false'}
              role="tab"
            > 
              {tab?.icon ? <ifx-icon icon = {tab.icon}></ifx-icon> : ''}
              {tab?.header}
            </li>
          ))}
          <div class="active-border"></div>
        </ul>
        <div class="tab-content">
          {Array.from(this.tabObjects).map((_, index) => (
            <div style={{ display: index === this.internalActiveTabIndex ? 'block' : 'none' }}>
              <slot name={`tab-${index}`} />
            </div>
          ))}
        </div>
      </div>
    );
  }
}

e2e test file:

import { newE2EPage } from '@stencil/core/testing';
describe('IfxTabs', () => {
  it('should render with tabs and change active tab', async () => {
    const page = await newE2EPage();
    await page.setContent(`
      <ifx-tabs>
        <ifx-tab header="Tab 1">
          <div>Tab 1 Content</div>
        </ifx-tab>
        <ifx-tab header="Tab 2">
          <div>Tab 2 Content</div>
        </ifx-tab>
        <ifx-tab header="Tab 3">
          <div>Tab 3 Content</div>
        </ifx-tab>
      </ifx-tabs>
    `);

    const tabs = await page.find('ifx-tabs');
    expect(tabs).toHaveClass('hydrated');

    const tabHeaders = await page.findAll('ifx-tabs >>> .tab-item');
    expect(tabHeaders.length).toBe(3);

    const tabContents = await page.findAll('ifx-tabs >>> .tab-content > div');
    expect(tabContents.length).toBe(3);

    const activeBorder = await page.find('ifx-tabs >>> .active-border');
    expect(activeBorder).toHaveClass('active-border');

    // Change active tab
    await tabHeaders[1].click();
    await page.waitForChanges();

    const newActiveBorder = await page.find('ifx-tabs >>> .active-border');
    expect(newActiveBorder).toHaveClass('active-border');

    const newActiveTabContent = await page.find('ifx-tabs >>> .tab-content > div[style="display: block;"]');
    expect(newActiveTabContent).not.toBeNull();

  });

  it('should set active tab when activeTabIndex is set', async () => {
    const page = await newE2EPage();
    await page.setContent(`
      <ifx-tabs active-tab-index="2">
        <ifx-tab header="Tab 1">
          Tab1Content
        </ifx-tab>
        <ifx-tab header="Tab 2">
          Tab2Content
        </ifx-tab>
        <ifx-tab header="Tab 3">
          Tab3Content
        </ifx-tab>
      </ifx-tabs>
    `);

    const activeTab = await page.find('ifx-tabs >>> .tab-item.active');

    const tabContents = await page.findAll('ifx-tabs >>> .tab-content > div');
    expect(tabContents.length).toBe(3);
    expect(await tabContents[0].isVisible()).toBe(false);
    expect(await tabContents[1].isVisible()).toBe(false);
    expect(await tabContents[2].isVisible()).toBe(true);

    expect(activeTab.innerText).toBe('Tab 3')    
  });

  
 

  it('should set last tab active when activeTabIndex out of bounds', async () => {
    const page = await newE2EPage();
    await page.setContent(`
      <ifx-tabs active-tab-index="3">
        <ifx-tab header="Tab 1">
          Tab1Content
        </ifx-tab>
        <ifx-tab header="Tab 2">
          Tab2Content
        </ifx-tab>
        <ifx-tab header="Tab 3">
          Tab3Content
        </ifx-tab>
      </ifx-tabs>
    `);


    const activeTab = await page.find('ifx-tabs >>> .tab-item.active');

    const tabContents = await page.findAll('ifx-tabs >>> .tab-content > div');
    expect(await tabContents[0].isVisible()).toBe(false);
    expect(await tabContents[1].isVisible()).toBe(false);
    expect(await tabContents[2].isVisible()).toBe(true);

    expect(activeTab.innerText).toBe('Tab 3')    
  });

  it('should set first tab active when activeTabIndex is smaller than 0', async () => {
    const page = await newE2EPage();
    await page.setContent(`
      <ifx-tabs active-tab-index="-1">
        <ifx-tab header="Tab 1">
          Tab1Content
        </ifx-tab>
        <ifx-tab header="Tab 2">
          Tab2Content
        </ifx-tab>
        <ifx-tab header="Tab 3">
          Tab3Content
        </ifx-tab>
      </ifx-tabs>
    `);


    const activeTab = await page.find('ifx-tabs >>> .tab-item.active');

    const tabContents = await page.findAll('ifx-tabs >>> .tab-content > div');
    expect(await tabContents[0].isVisible()).toBe(true);
    expect(await tabContents[1].isVisible()).toBe(false);
    expect(await tabContents[2].isVisible()).toBe(false);

    expect(activeTab.innerText).toBe('Tab 1')    
  });

});

Error:

IfxTabs › should set first tab active when activeTabIndex is smaller than 0
App did not load within 10000ms. Please ensure the content loads a stencil application.

Example of a test that passes:

import { newE2EPage } from '@stencil/core/testing';

describe('ifx-accordion-item', () => {
  let page;
  let element;

  beforeEach(async () => {
    page = await newE2EPage();
    await page.setContent('<ifx-accordion-item caption="Test"></ifx-accordion-item>');

    element = await page.find('ifx-accordion-item');
  });

  it('renders', async () => {
    expect(element).toHaveClass('hydrated');
  });

  it('displays the caption', async () => {
    const captionElement = await page.find('ifx-accordion-item >>> .accordion-caption');
    expect(captionElement.innerText).toEqual('Test');
  });

  it('expands and collapses on click', async () => {
    const titleElement = await page.find('ifx-accordion-item >>> .accordion-title');

    // Initially closed
    expect(await element.getProperty('open')).toBeFalsy();

    await titleElement.click();

    // Open after first click
    expect(await element.getProperty('open')).toBeTruthy();

    await titleElement.click();

    // Closed after second click
    expect(await element.getProperty('open')).toBeFalsy();
  });
});

Not sure if relevant, but when I push my changes to github, github-actions runs the tests, and often there are less tests that fail on github actions than locally. For example just now, the e2e checks failed on github and the log shows 8 tests failed, while locally it's 12.

Also, with Stencil version ^4.22.3, and Puppeteer version 21.9.0, all tests were passing locally, but some kept failing on GitHub actions (even after deleting cache)

Update: now even with Stencil version 4.22.3 and Puppeteer version 21.9.0 tests fail too with the same error..

[johnjenkins]

To those affected, I believe there is a conflation of a couple of issues here.

1. The initially reported (either App did not load in allowed time or Please ensure the content loads a stencil application)

This was addressed via #6098 / v4.24.0

2. App did not load within 10000ms - A secondary issue (introduced by fix(testing): update puppeteer, default to 'new' headless #6098) which changed the default timeout for all e2e tests from 30s to 10s

The reverting of that change is merged and awaiting release. If you can't wait, you can set the timeout on the newE2EPage({... timeout: 30000 })

[tishoyanchev]

To those affected, I believe there is a conflation of a couple of issues here.

1. The initially reported (either App did not load in allowed time or Please ensure the content loads a stencil application)

This was addressed via #6098 / v4.24.0

2. App did not load within 10000ms - A secondary issue (introduced by fix(testing): update puppeteer, default to 'new' headless #6098) which changed the default timeout for all e2e tests from 30s to 10s

The reverting of that change is merged and awaiting release. If you can't wait, you can set the timeout on the newE2EPage({... timeout: 30000 })

Do you guys have any time frame for the 2nd issue?