Wait for resource to complete - ruby

I have a recipe that looks similar to this:
...
custom_resource1 "example" do
writing stuff to a file
end
log 'File found!' do
message "Found it
level :info
notifies :run, 'custom_resource2[example]', :immediately
only_if { ::File.exists?(file) }
end
...
custom_resource1 is a big resource with other resources inside, and takes some time to complete (iterates over some data_bags and writes to a file).
Sometimes, I see that custom_resource1 fails during a chef run, but still custom_resource2 is triggered before the recipe fails.
Is there any way to ensure that custom_resource1 either failed or completed before moving on?

That isn't possible, Chef uses an entirely blocking execution model (other than the two-pass loading system). The full action method for each resource is run in order with no concurrency. You would have to post more code to isolate the actual problem.

I also thought it was strange, because the log statement was never printed out, even though custom_resource2 were triggered by the notify. The soloution was to remove the log statement and instead add:
custom_resource2 "example" do
do stuff
only_if { ::File.exists?(file) }
end
Guess it has something to do with the different chef phases

Related

How to write a spec for trap handling on rspec?

say I have a class which traps SIGTERM and I want to write a spec to verify that that specific code is ran when SIGTERM is received. What is the proper way of doing it?
I've followed the answer for this topic: How to test signal handling in RSpec, particularly handling of SIGTERM?, but rspec is terminated on Process.kill happens.
I've also tried it like this:
raise SignalException.new('TERM')
But it doesn't seem to do anything (trap is not triggered). Finnaly, I've tried using 'allow' to substitute a method which is called during the spec to raise the signal or call Process.kill like this:
allow(<Class>).to receive(<method>).and_raise(SignalException.new('TERM'))
allow(<Class>).to receive(<method>).and_return(Process.kill 'TERM',0)
When raising the signal it also doesn't seem to do anything, and calling Process.kill simply ends rspec without a stack trace, just the word 'Terminated').
The trap code is like this:
trap('SIGTERM') {
Rails.logger.error('term')
#received_sigterm = true
}

Activiti SubProcess Signal

I have activiti process, that does not seem to work the way I expect it to.
Process is pretty straightforward:
Once process is started, it starts subprocess.
Subprocess has parallel gateway, that branches to signal catching event and user tasks.
Once singal is caught, subprocess throws error (end error event).
While signal is not caught, user can complete tasks.
Then main process catches that error and ends process.
My problem: I start 4 such processes, that use 1 singal. When this signal is sent, all 4 of them should be ended. But they are not. It seems they are ended in random order? Somtimes 1 or 2 are ended, but never all of them...I get no errors/exceptions and have no idea what is wrong...
It would seem that some processes just wont catch the signal that is sent to them. Why could that be so?
Process schema:
Have you added logging to your "check caught signal conditions" to determine if all signals actually trigger this step?
Also, you can query signal subscriptions to determine if you have as many as you expect:
List executions = runtimeService.createExecutionQuery()
.signalEventSubscriptionName("alert")
.list();
I expect either your check logic is not processing correctly or you dont have a "token" waiting at the signal.
Greg
I have created simple jUnit test based on the already existing signal boundary event tests in flowable project (should behave in the same way as activiti)
#Deployment(resources = "org/activiti/engine/test/bpmn/event/signal/SignalEventTest.testSignalBoundaryOnSubProcess.bpmn20.xml")
public void testMultipleCatchEventsOnSubProcess() {
ProcessInstance processInstance1 = runtimeService.startProcessInstanceByKey("signalEventOnSubprocess");
ProcessInstance processInstance2 = runtimeService.startProcessInstanceByKey("signalEventOnSubprocess");
ProcessInstance processInstance3 = runtimeService.startProcessInstanceByKey("signalEventOnSubprocess");
assertThat("There must be three subscriptions to the signal.", createEventSubscriptionQuery().count(), is(3L));
runtimeService.signalEventReceived("stopSignal");
assertProcessEnded(processInstance1.getProcessInstanceId());
assertProcessEnded(processInstance2.getProcessInstanceId());
assertProcessEnded(processInstance3.getProcessInstanceId());
}
The test passes. Could you create failing jUnit test to reproduce the issue (with the dependencies)?
Martin

Timeout in a delayed job

I have some code that potentially can run for a longer period of time. However if it does I want to kill it, here is what I'm doing at the moment :
def perform
Timeout.timeout(ENV['JOB_TIMEOUT'].to_i, Exceptions::WorkerTimeout) { do_perform }
end
private
def do_perform
...some code...
end
Where JOB_TIMEOUT is an environment variable with value such as 10.seconds. I've got reports that this still doesn't prevent my job from running longer that it should.
Is there a better way to do this?
I believe delayed_job does some exception handling voodoo with multiple retries etc, not to mention that I think do_perform will return immediately and the job will continue as usual in another thread. I would imagine a better approach is doing flow control inside the worker
def perform
# A nil timeout will continue with no timeout, protect against unset ENV
timeout = (ENV['JOB_TIMEOUT'] || 10).to_i
do_stuff
begin
Timeout.timeout(timeout) { do_long_running_stuff }
rescue Timeout::Error
clean_up_after_self
notify_business_logic_of_failure
end
end
This will work. Added benefits are not coupling delayed_job so tightly with your business logic - this code can be ported to any other job queueing system unmodified.

Can ruby exceptions be handled asynchronously outside of a Thread::handle_interrupt block?

At first glance, I thought the new ruby 2.0 Thread.handle_interrupt was going to solve all my asynchronous interrupt problems, but unless I'm mistaken I can't get it to do what I want (my question is at the end and in the title).
From the documentation, I can see how I can avoid receiving interrupts in a certain block, deferring them to another block. Here's an example program:
duration = ARGV.shift.to_i
t = Thread.new do
Thread.handle_interrupt(RuntimeError => :never) do
5.times { putc '-'; sleep 1 }
Thread.handle_interrupt(RuntimeError => :immediate) do
begin
5.times { putc '+'; sleep 1}
rescue
puts "received #{$!}"
end
end
end
end
sleep duration
puts "sending"
t.raise "Ka-boom!"
if t.join(20 + duration).nil?
raise "thread failed to join"
end
When run with argument 2 it outputs something like this:
--sending-
--received Ka-boom!
That is, the main thread sends a RuntimeError to the other thread after two seconds, but that thread doesn't handle it until it gets into the inner Thread.handle_interrupt block.
Unfortunately, I don't see how this can help me if I don't know where my thread is getting created, because I can't wrap everything it does in a block. For example, in Rails, what would I wrap the Thread.handle_interrupt or begin...rescue...end blocks around? And wouldn't this differ depending on what webserver is running?
What I was hoping for is a way to register a handler, like the way Kernel.trap works. Namely, I'd like to specify handling code that's context-independent that will handle all exceptions of a certain type:
register_handler_for(SomeExceptionClass) do
... # handle the exception
end
What precipitated this question was how the RabbitMQ gem, bunny sends connection-level errors to the thread that opened the Bunny::Session using Thread#raise. These exceptions could end up anywhere and all I want to do is log them, flag that the connection is unavailable, and continue on my way.
Ideas?
Ruby provides for this with the ruby Queueobject (not to be confused with an AMQP queue). It would be nice if Bunny required you to create a ruby Queue before opening a Bunny::Session, and you passed it that Queue object, to which it would send connection-level errors instead of using Thread#raise to send it back to where ever. You could then simply provide your own Thread to consume messages through the Queue.
It might be worth looking inside the RabbitMQ gem code to see if you could do this, or asking the maintainers of that gem about it.
In Rails this is not likely to work unless you can establish a server-wide thread to consume from the ruby Queue, which of course would be web server specific. I don't see how you can do this from within a short-lived object, e.g. code for a Rails view, where the threads are reused but Bunny doesn't know that (or care).

Using django-celery chord, celery.chord_unlock keeps executing forever not calling the provided callback

I'm using Django Celery with Redis to run a few tasks like this:
header = [
tasks.invalidate_user.subtask(args = (user)),
tasks.invalidate_details.subtask(args = (user))
]
callback = tasks.rebuild.subtask()
chord(header)(callback)
So basically the same as stated in documentation.
My problem is, that when this task chord is called, celery.chord_unlock task keeps retrying forever. Tasks in header finish successfully, but because of chord_unlock never being done, callback is never called.
Guessing that my problem is with not being able to detect that the tasks from header are finished, I turned to documentation to look how can this be customized. I've found a section, describing how the synchronization is implemented, there is an example provided, what I'm missing is how do I get that example function to be called (i.e. is there a signal for this?).
Further there's a note that this method is not used with Redis backend:
This is used by all result backends except Redis and Memcached, which increment a counter after each task in the header, then applying the callback when the counter exceeds the number of tasks in the set.
But also says, that Redis approach is better:
The Redis and Memcached approach is a much better solution
What approach is that? How is it implemented?
So, why is chord_unlock never done and how can I make it detect finished header tasks?
I'm using: Django 1.4, celery 2.5.3, django-celery 2.5.5, redis 2.4.12
You don't have an example of your tasks, but I had the same problem and my solution might apply.
I had ignore_result=True on the tasks that I was adding to a chord, defined like so:
#task(ignore_result=True)
Apparently ignoring the result makes it so that the chord_unlock task doesn't know they're complete. After I removed ignore_result (even if the task only returns true) the chord called the callback properly.
I had the same error, I changed the broker to rabbitmq and chord_unlock is working until my task finishes (2-3 minutes tasks)
when using redis the task finishes and chord_unlock only retried like 8-10 times every 1s, so callback was not executing correctly.
[2012-08-24 16:31:05,804: INFO/MainProcess] Task celery.chord_unlock[5a46e8ac-de40-484f-8dc1-7cf01693df7a] retry: Retry in 1s
[2012-08-24 16:31:06,817: INFO/MainProcess] Got task from broker: celery.chord_unlock[5a46e8ac-de40-484f-8dc1-7cf01693df7a] eta:[2012-08-24 16:31:07.815719-05:00]
... just like 8-10 times....
changing broker worked for me, now I am testing #Chris solution and my callback function never receives the results from the header subtasks :S, so, it does not works for me.
celery==3.0.6
django==1.4
django-celery==3.0.6
redis==2.6
broker: redis-2.4.16 on Mac OS X
This could cause a problem such that; From the documentation;
Note:
If you are using chords with the Redis result backend and also overriding the Task.after_return() method, you need to make sure to call the super method or else the chord callback will not be applied.
def after_return(self, *args, **kwargs):
do_something()
super(MyTask, self).after_return(*args, **kwargs)
As my understanding, If you have overwritten after_return function in your task, it must be removed or at least calling super one.
Bottom of the topic:http://celery.readthedocs.org/en/latest/userguide/canvas.html#important-notes

Resources