Skip to content

Operator crash after update to v0.7.0 #583

Closed
#592
Closed
Operator crash after update to v0.7.0#583
#592
Labels
bugSomething isn't workingcontroller operationsDealing with the reconcile-loop and other internal aspects of the various Solr operator controllers
Milestone
v0.7.1

Description

@angelprada
angelprada
opened on Jul 4, 2023

Hello.

I've tried to upgrade solr-operator from v0.6.0 to v0.7.0 in AWS EKS 1.27 but it crashes constantly:

2023-07-03T08:49:18Z	INFO	setup	solr-operator Version: v0.7.0
2023-07-03T08:49:18Z	INFO	setup	solr-operator Git SHA: 0e16c6b
2023-07-03T08:49:18Z	INFO	setup	solr-operator Build Time: Mon Apr 24 19:23:35 UTC 2023
2023-07-03T08:49:18Z	INFO	setup	Go Version: go1.19.8
2023-07-03T08:49:18Z	INFO	setup	Go OS/Arch: linux / amd64
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12ab950]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:88
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc00004f560, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:69 +0x5f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc00004e3f0, 0x15}, {0xc000049060, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:35 +0x2f8
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x1815ff3?)
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:310 +0x47c
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc000001e00?)
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:198 +0x5c
k8s.io/client-go/discovery.ServerGroupsAndResources({0x1a8a320, 0xc0002b32f0})
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:392 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:356 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc0004c50d0)
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:621 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
	/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:355 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x1a8a320?, 0xc0002b32f0?})
	/go/pkg/mod/k8s.io/client-go@v0.26.1/restmapper/discovery.go:148 +0x42
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/client/apiutil/dynamicrestmapper.go:86 +0x25
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/client/apiutil/dynamicrestmapper.go:119
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc000358870?, {0x0, 0x0, 0x2fb3e8039a338f01?})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/client/apiutil/dynamicrestmapper.go:99 +0x176
sigs.k8s.io/controller-runtime/pkg/cluster.setOptionsDefaults.func1(0xc00011d6c0?)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/cluster/cluster.go:217 +0x25
sigs.k8s.io/controller-runtime/pkg/cluster.New(0xc0001c3200, {0xc000143a58, 0x1, 0x0?})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/cluster/cluster.go:159 +0x18d
sigs.k8s.io/controller-runtime/pkg/manager.New(_, {0xc00011d6c0, 0x0, 0x0, {{0x1a87000, 0xc00012f1c0}, 0x0}, 0x1, {0x0, 0x0}, ...})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/manager/manager.go:351 +0xf9
main.main()
	/workspace/main.go:171 +0x7b5

Activity

  1. HoustonPutman commented on Jul 5, 2023

    @HoustonPutman
    HoustonPutman
    on Jul 5, 2023
    Contributor

    Unfortunately I can't replicate this on a KinD cluster running 1.27, but I did find this when googling:

    elastic/cloud-on-k8s#6848

    So maybe we just need to update the version of the kubernetes client for the operator.

  2. angelprada commented on Jul 5, 2023

    @angelprada
    angelprada
    on Jul 5, 2023
    Author

    Hello.

    We had similar problems with external-secrets external-secrets/external-secrets#2344

    Hope it helps.

  3. HoustonPutman
    added this to the v0.7.1 milestone on Jul 17, 2023
  4. HoustonPutman
    added
    bugSomething isn't working
    controller operationsDealing with the reconcile-loop and other internal aspects of the various Solr operator controllers
    on Jul 17, 2023
  5. HoustonPutman
    closed this as completedin #592on Jul 17, 2023

  6. HoustonPutman commented on Jul 17, 2023

    @HoustonPutman
    HoustonPutman
    on Jul 17, 2023
    Contributor

    Fixed this. I'll start a v0.7.1 release soon.

  7. HoustonPutman commented on Jul 17, 2023

    @HoustonPutman
    HoustonPutman
    on Jul 17, 2023
    Contributor

    @angelprada The release candidate is out to try, please let me know if the docker image apache/solr-operator:v0.7.1-rc1 works for you. It should be a drop-in replacement for v0.7.0.

  8. angelprada commented on Jul 17, 2023

    @angelprada
    angelprada
    on Jul 17, 2023
    Author

    @HoustonPutman Tried and worked as expected!

    2023-07-17T17:46:46Z	INFO	setup	solr-operator Version: v0.7.1
2023-07-17T17:46:46Z	INFO	setup	solr-operator Git SHA: 5205b93
2023-07-17T17:46:46Z	INFO	setup	solr-operator Build Time: Mon Jul 17 17:08:17 UTC 2023
2023-07-17T17:46:46Z	INFO	setup	Go Version: go1.19.11
2023-07-17T17:46:46Z	INFO	setup	Go OS/Arch: linux / amd64
2023-07-17T17:46:47Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
2023-07-17T17:46:47Z	INFO	setup	starting manager
2023-07-17T17:46:47Z	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-07-17T17:46:47Z	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
I0717 17:46:47.169060       1 leaderelection.go:248] attempting to acquire leader lease kube-system/88488bdc.solr.apache.org...
I0717 17:47:17.922432       1 leaderelection.go:258] successfully acquired lease kube-system/88488bdc.solr.apache.org

    Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcontroller operationsDealing with the reconcile-loop and other internal aspects of the various Solr operator controllers

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions